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CCSSO/AIR/WCER 

Longitudinal Study of the Effects of Professional Development on Improving 
Mathematics and Science Instruction (MSP PD Study) 

Year 2 progress report 

Math-Science Partnership (MSP) RETA Grant (EHR-0233505) 
National Seienee Foundation 



Executive Summary 

In fall 2002 the National Science Foundation (NSF) announced 24 major grants to 
establish Mathematies and Science Edueation Partnership (MSP) programs across the 
U.S. The overall program objective of these grants is to “increase the capacity of preK-12 
educational systems and institutions of higher education to provide the requisites for 
learning to high standards in mathematics and science, and particularly to reduce the 
aehievement gaps among student populations.” One of the specifie goals of MSP is “to 
contribute to the national capacity to engage in large-scale reform through participation in 
a network of researchers and praetitioners, organized through the MSP program, that will 
study and evaluate educational reform and experimental approaches to the improvement 
of teaeher preparation and professional development (Goal 3, NSF 02-061 program 
announcement) . ’ ’ 

The present report summarizes progress after two years of a three- year study under the 
MSP RETA program. The study is being conducted by a research team eomprised of staff 
from the Council of Chief State School Officers, American Institutes for Research, and 
Wisconsin Center for Edueation Research. 

Testing Effectiveness of Survey Tools in Evaluation 

One purpose of the MSP PD study is to test the use of teaeher self-report survey and web- 
based tools for collecting, analyzing and reporting data on the quality of professional 
development, and the usefulness of these data tools for evaluating effeets of professional 
development. The work is being conducted in four MSP sites from the first cohort of 
MSP grant sites (starting Fall 2002) — two comprehensive projeets and two targeted 
projects. 



• After the seeond year of three-year study design, we have demonstrated the use 
of the Surveys of Enacted Curriculum (SEC) for analyzing differences in instructional 
practices and content of math and science teaching. Teachers across the four MSP sites 
eompleted the surveys at an overall response rate greater than 80 pereent. We 
demonstrate in this Report the use of these data for measuring differences in instruction at 
the baseline year. We also demonstrated the use of the SEC, a paper-and-pencil survey, 
and a Web-based monthly log for reporting on differences in the quality and distribution 
of professional development. 



1 




MSP PD Study: Yr 2 Report 



• The study has developed, tested, and demonstrated the use of a web-based monthly 
Professional Development Activity Log (PDAL) system to regularly gather data from 
teachers on the professional development activities. The PDAL system had response 
rates as high as 62 percent in a single month, although the average rate was 57 percent in 
a single month; many teachers however, did not complete all months of the PDAL. The 
study has tested several methods of increasing response rates including online and CD 
training packages, postcards, incentive pay, and phone follow-ups. We will report on 
ways to improve PDAL response rates in the project’s final report. 

Preliminary Evidence of Effectiveness of MSP Professional Development 

The study is focusing on a sample of teachers targeted for MSP participation and non- 
participating comparison teachers in each MSP site. We found some empirical evidence 
that MSP initiatives are achieving their intended purpose of providing quality 
professional development activities for math and science teachers. For example, the four 
MSP program sites have instituted some of the structural features of high quality 
professional development activities identified in research literature, such as extended 
contact hours and prolonged activity span. These are among the structural features likely 
to set the conditions for teachers to receive intense, sustained, and in-depth learning 
opportunities, which, in turn, are expected to produce effects on teaching practice. 
Furthermore, we found that teachers targeted for participation in MSP programs tend to 
experience a stronger content focus in their professional development, particularly in 
science. Combined with sustained learning opportunities, the stronger content focus 
afforded by MSP programs is expected to allow teachers to incorporate new content and 
instructional strategies in their classrooms. 

When the follow-up SEC data are made available in spring of 2005, we will test the most 
interesting and important hypothesis of this study: “after a year of MSP implementation, 
we expect teachers in the treatment group (i.e., teachers targeted for MSP participation) 
to exhibit higher quality instruction, as measured by the alignment of instruction with 
state content standards, than teachers in the comparison group, after controlling for any 
year 1 differences.” Our analysis of the effects of professional development is based on a 
quasi-experimental analysis of teachers targeted for participation in MSP (for simplicity 
termed “treatment” teachers), and a group of non-participating comparison teachers, 
because random assignment of teachers to MSP and comparison groups was not feasible. 
Our analysis of the effects of professional development will be facilitated by our 
longitudinal study design, the collection of extensive baseline data to serve as potential 
control variables, and the use of such innovative instruments as the PDAL and SEC 
funded by NSE’s RETA grant program. 



2 




MSP PD Study: Yr 2 Report 



I. Goals OF Study 



Introduction 

In fall 2002 the National Science Foundation (NSF) announced 24 major grants to 
establish Mathematics and Science Education Partnership (MSP) programs across the 
U.S. The overall program objective of these grants is to “increase the capacity of preK-12 
educational systems and institutions of higher education to provide the requisites for 
learning to high standards in mathematics and science, and particularly to reduce the 
achievement gaps among student populations.” One of the specific goals of MSP is “to 
contribute to the national capacity to engage in large-scale reform through participation in 
a network of researchers and practitioners, organized through the MSP program, that will 
study and evaluate educational reform and experimental approaches to the improvement 
of teacher preparation and professional development (Goal 3, NSF 02-061 program 
announcement).” 

The present report summarizes progress after two years of a three-year study under the 
MSP RETA program. The study is being conducted by a research team comprised of staff 
from the Council of Chief State School Officers, American Institutes for Research, and 
Wisconsin Center for Education Research. 

Organization of the Report 

This progress report is organized in six main sections. Eirst, in this section, we discuss 
the goals of the study. Then, in Section II, we turn to the study design and methodology. 
Section III describes data from the baseline survey of teachers, collected in year 1 of the 
study using the Survey of the Enacted Curriculum. Section IV describes initial data from 
the Professional Development Activity Log, collected in year 2 of the study. Section V 
illustrates program site-specific effects by using a single site. Finally, we summarize the 
initial findings of the study. 

Research Questions 

To assist NSF and the Math-Science Partnerships with the goal of improving capacity for 
evaluation of the models for improving teacher knowledge and skills, the CCSSO/AIR/ 
WCER team is conducting a three-year empirical study that is testing an objective, 
reliable methodology for measuring the quality of professional development activities 
and using the data obtained to examine the effects professional development on 
improving the quality of instruction in mathematics and science education. More 
specifically, the study has three main research questions: 

To what extent is the quality of the professional development supported by MSP 
activities consistent with research-based definitions of quality? 

What effects do teachers’ professional development experiences have on instructional 
practices and content taught in math and science classes? Are high-quality 
professional development activities more likely than lower-quality activities to 
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increase the alignment of instructional content with state standards and 
assessments? 

How can MSP projects use study findings and research tools tested in the study to 
improve professional development and evaluation based on measuring 
improvement in math and science instruction? 

In this year 2 progress report, we describe the design and implementation of the study and 
some of the early findings regarding the first two research questions. In the final report at 
end of year three we will fully address all three research questions. 

Rationale for the Study: Developing and Testing a Model for Evaluation 

For decades, educators and policy-makers have seen statistics that demonstrate the 
lackluster performance of American students in the areas of mathematics and science. 
Despite years of efforts to improve both instructional practice as well as students’ 
outcomes, there is still much room for further improvement. Recent results from the 
National Assessment of Educational Progress (NAEP) in mathematics and science show 
that although scores have improved in the 1990s, a majority of our students score below 
the proficient level in mathematics and science. In addition, the results from the Third 
International Mathematics and Science Study highlight the problems of wide variation in 
student performance in mathematics and science across our schools and the declining 
performance of U.S. students in the higher grades relative to other systems. 

The TIMSS findings provide strong evidence that predominant teaching practices do not 
enable students to acquire the understanding or flexible skills for problem solving in 
mathematics or science. Of equal concern is the persistent gap between the achievement 
levels of poor and minority students with their more advantaged peers. The No Child 
Left Behind Act/ H.R. 1 codifies the national goal of closing the achievement gap of 
poor, minority, and limited English students with more advantaged students in our 
schools. 

One major strategy for improving student performance and reducing the achievement gap 
is to set challenging content standards for all students. This strategy reflects a new kind 
of equity in education — one that has slowly shifted from equality of educational inputs to 
equality of educational outputs. But achieving this form of educational equity requires 
fundamental changes in what students are taught, and how they are taught. Education 
reforms, if they are to improve student achievement, must first change instructional 
practice at the classroom and school levels, and recent research has shown that one of the 
most powerful explanatory variables of the achievement gap between majority and 
minority students is the content of instruction. 

The National Commission on Mathematics and Science Teaching for the 21st Century 
recommended in 2000 that the U.S.— a) establish an ongoing system to improve the 
quality of mathematics and science teaching in grades K-12; and b) increase significantly 
the number of mathematics and science teachers and improve the quality of their 
preparation especially for teachers of low-income students. Clearly, many of our current 
science and mathematics teachers did not receive an adequate initial preparation for 
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teaching. These teachers need continuing professional development in math-science 
content and pedagogy. 

The Math-Science Partnership grants will serve as an important tool and catalyst in 
improving instructional practice in mathematics and science. The Partnership projects 
will bring together school districts, institutions of higher education, scientists and 
mathematicians, and professional organizations to promote world-class science and 
mathematics education for all students. MSP projects will emphasize a vital role for 
mathematicians, scientists and engineers in enhancing the capacity of the teacher 
workforce to provide challenging curriculum content. This will entail pre-service and in- 
service professional development of teachers as a central element of each MSP plan. 

The MSP strategy clearly builds upon more than a decade of NSF support for and 
leadership in systemic reform. The goal of systemic initiatives in math and science of the 
1990 s was to ensure that efforts toward improvement were coherent, aligned, and 
consistent with challenging standards or curriculum frameworks for teaching and 
learning. However, evidence is still inconclusive regarding effects of professional 
development efforts in significantly changing instruction in math and science. 

The CCSSO/AIRAVCER evaluation study will directly address the lack of large-scale, 
empirical evidence on the effects of professional development on mathematics and 
science instruction. This longitudinal study will measure the degree to which districts 
implement high-quality professional development, through MSP support, and examine 
the effects of such professional development on teachers’ classroom practice. The study 
will also test the reliability of teacher surveys and logs for collecting, analyzing and 
reporting data on key characteristics of professional development and subsequent 
instructional practices and their alignment to state standards. Ultimately, the study should 
help evaluators, policy-makers and educators to better evaluate the effectiveness of 
professional development activities that teachers experience. 

Study Builds on Recent Development of Data Collection and Analysis 
Tools 

To achieve its goals, the study relies on the recent development of tools for measuring the 
key variables — alignment of instruction with standards and assessments, and the quality 
of professional development activities. 

Surveys of Enacted Curriculum measure alignment of instruction with 
standards and assessments. If poor and minority children are to receive a high quality, 
standards-based education - and ultimately to reduce the achievement gap - then the 
instruction they receive must be aligned with the state content standards. Hence, a key 
element in understanding the impact of standards-based reform on student achievement is 
a measure of the alignment between the curricular content to which students are exposed 
and the content standards the state and district hope to implement. Indeed, this is a 
central element of this study of the effects of MSP professional development: 
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instructional quality is defined and measured as the degree of alignment between 
classroom instruction and standards 

The MSP study builds on the advances in design and application of Surveys of Enacted 
Curriculum in math and science (Blank, Porter, Smithson, 2001; Porter, 2002; Blank, 
2002). Teachers complete surveys which ask them to report on subject content and 
practices used in one eourse/grade during a sehool year, and time allocated to different 
instructional practices. Prior grants from NSF have also resulted in unique data reporting 
methods that are effeetive for various users and audiences, employing scales, item 
profiles, graphs, and content maps. The recent experimental design study supported by 
NSF, Data on Enacted Currieulum, tested the use of the currieulum data reports in 
improving instruction in math and science (see Blank, 2004, Data on Enacted 
Curriculum (DEC) Study: Summary of Eindings). 

The Surveys provide in-depth information on instructional content using a two- 
dimensional matrix design. The two dimensions are: (a) Topic Area, including more 
fine-grained subtopies, and (b) Expectations for Students, with a foeus on the cognitive 
demand. Teachers are asked to report the amount of time spent on topics and then the 
expeetations that are emphasized for the topics taught. [See further explanation of the 
Content Matrix in the Supplementary Materials.] 

One important benefit of the Surveys of Enacted Curriculum is that the two-dimensional 
content matrix is used to analyze the content included in standards and assessments, as 
well as the content teachers cover in class, making it possible to compute an objective 
measure of alignment. Content coding and alignment analysis is aceomplished through 
procedures developed and tested by Porter and Smithson (2001; Gamoran, et al, 1997). 

Recent research identifies criteria of quality professional development. Over 
the past decade, a large body of literature has emerged on effective professional 
development, teacher learning, and teacher change (Carey & Frechtling, 1997; Cohen & 
Hill, 1998; Kennedy, 1998; Loueks-Horsley et al., 1998; Richardson & Placier, 2002). 
Despite the amount of research, relatively little systematic research has been conducted 
on the effects of different professional development programs on improving teaehing or 
on improving student outcomes (see Kennedy, 1998; Supovitz, 2002). 

A research team at AIR recently completed the National Evaluation of the Eisenhower 
Professional Development Program and developed a model to analyze the relationship 
between features of professional development and teachers’ self-reported increases in 
knowledge and skills, and changes in teaching practice (Birman, et al, 2002; Desimone, 
2002). On the basis of national data, AIR concluded that six key features of professional 
development are effeetive in improving teaching practice. Three are characteristics of the 
substance of the activity: (1) the degree to which the activity has active learning 
opportunities for teaehers, (2) the extent to whieh the aetivity has a content focus on 
mathematics or science, and (3) the degree to which the activity promotes coherence in 
teachers’ professional development by incorporating experiences that are consistent with 
teachers’ goals and aligned with state standards and assessments (see Garet, et al., 1999; 
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Caret et al., 2001).The remaining three features are characteristics of the structure of the 
activity, that facilitate the core, substantive features: (4) the organization of the activity — 
whether it is a reform type such as a study group or teacher network, in contrast to a 
traditional workshop or conference, (5) the duration of the activity, including the total 
number of contact hours and the span of time over which it extends, and (6) the extent to 
which the activity has collective participation of groups of teachers from the same 
school, department, or grade. For the present study, these characteristics of quality were 
the foundation for developing a web-based Professional Development Activity Log 
(PDAL) to measure the features of the professional development in which teachers 
participate and to examine the effects of professional development on instruction. 

Section II of the Year 2 report provides description of the study design. Section II briefly 
outlines the initial data from the baseline survey, and section IV summarizes the data 
from the monthly logs with teachers. 
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II. Study Design and Methods 

The CCSSO/AIRAVCER team’s longitudinal study design has three main steps: 

(1) In year 1, prior to start of MSP professional development activities, develop 
and refine the data collection instruments, and then collect baseline data through teacher 
surveys to measure the instructional content and pedagogy teachers employ in 
mathematics and science instruction; 

(2) In year 2, measure the characteristics of the professional development 
activities in which teachers participate through MSP; and 

(3) In year 3, re-examine instructional practices, to determine if teachers’ practice 
had indeed changed after participation in MSP-supported activities, and analyze the 
change in practices in relationship to degree of alignment with state content standards. A 
key contribution of the study is the measure of instruction both before and after 
professional development, and the analysis of improvement based on the goal of 
alignment with standards. 

Exhibit 1 provides a graphic of the key tasks and scheduled activities for the three-year 
(2002-05) study design. 
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Exhibit 1 outlines the conceptual framework for the study and highlights key data 
collection activities and time frames. The initial study design included four selected MSP 
projects from the first cohort of NSF MSP awards (Fall 2002): 

• Brockport/Rochester, NY 

• Cleveland, OH 

• Corpus Christ! area, TX 

• El Paso area, TX. 

The projects were selected based on their MSP design, which was to provide professional 
development to middle school math and science teachers during the first year — allowing 
for measurement of change over time within a three-year scope of the study. A fifth MSP 
project. New Jersey, decided to use the Surveys of Enacted Curriculum as part of its 
evaluation design and collected data from math and science teachers in the 12-district 
target area of the study 

Ideally, to obtain unbiased estimates of the effects of MSP professional development on 
instruction, teachers would be randomly assigned to participate in MSP or to serve as a 
control group. Given the fact that the MSP projects were already underway at the time 
our study began, random assignment was not feasible. Thus, the study uses a quasi- 
experimental design. Within each site, at the start of the baseline year (year 1), we 
worked with site staff to identify a group of teachers targeted for participation in MSP 
activities, and a matched set of comparison teachers not selected for participation. 

The specific approach to matching the targeted and comparison teachers varied across the 
four sites, depending on the way each site organized its MSP efforts. In Brockport (site 
I), the MSP project sought applicants for participation in MSP activities. We identified 
applicants who were selected for participation as the treatment group, and applicants not 
selected as the comparison group. In Cleveland (site 2), the MSP project selected 
teachers for participation in MSP activities, and identified math and science non- 
participating teachers to serve as a comparison group. In Corpus Christ! area (site 3), all 
math teachers participating in MSP professional development in year 1 were selected as 
the treatment group, and other math and science teachers in the target schools are the 
comparison group. Finally, in El Paso area (site 4), the MSP project is using a school- 
based approach to the delivery of professional development. The project identified six 
middle schools for participation in MSP and identified five other schools located in the 
same districts as comparison schools. 

In the text that follows, we refer to the teachers targeted for MSP participation in each 
site as the treatment group, and we refer to the teachers identified as comparisons as the 
comparison group. The nature of the professional development “treatment” received by 
teachers of course varied from site to site, and also within site. Thus our use of the term 
“treatment” should not be interpreted to mean that there is a single, standardized “MSP 
treatment.” Our use of the term “treatment” is simply a short-hand for “teachers targeted 
for MSP participation.” Because our treatment and comparison teachers were not 
randomly assigned, we have measured a large number of teacher characteristics at 
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baseline (year 1), to use as potential control variables in our analyses of the effects of 
professional development, including teacher background, prior instruction, and 
participation in professional development. We believe these variables should help reduce 
potential biases in the analysis. But we cannot rule out the possibility that teachers in the 
treatment and comparison groups differ in unmeasured ways - for example, in 
motivation. Thus, conclusions about the effects of professional development should be 
interpreted as potential effects; a more rigorous design would be required to draw causal 
conclusions. 

Study hypotheses 

In the study, we examine six hypotheses: 

Hypothesis 1: In year 1, prior to the initiation of MSP, we expect sites to 
differ in the quality of the professional development teachers experienced and in the 
quality of classroom instruction as measured hy the alignment with state content 
standards. We anticipate that differences in state and district policies, resources, and 
history will be reflected in differences across sites in professional development and 
instruction, as measured by the Wave 1 SEC. 

Hypothesis 2: In year 1, we expect teachers in the treatment group (i.e., 
teachers targeted for MSP participation) to be similar to comparison teachers in 
background, in the quality of the professional development they experienced, and in 
the quality of classroom instruction they provided. While we were unable to use 
random assignment methods to allocate teachers to treatment and control conditions, we 
worked with each of the four MSP sites to create as closely matched a set of treatment 
and comparison teachers as possible. 

Hypothesis 3: In year 2, the initial year of MSP implementation, we expect 
sites to differ in the quality of the professional development teachers experienced. 

We anticipate differences across sites in the quality of professional development during 
year 2, in part due to variation in state and district policies, resources, and history. In 
addition, we anticipate variation across sites in MSP professional development, since 
each local university-school district partnership adopted and implemented different 
strategies to achieve their MSP program goals, based in part on local curricula, 
knowledge and skills. 

Hypothesis 4: In year 2, we expect that teachers in the treatment group (i.e., 
teachers targeted for MSP participation) experienced higher quality professional 
development than teachers in the comparison group. If MSP initiatives are 
effectively implemented in each of the program sites, we expect that teachers in the 
treatment group will show evidence of receiving professional development that is higher 
in quality than teachers in the comparison group (e.g., they will experience more 
opportunities for active learning, as well as activities that are more intensive and 
coherent). 
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Hypothesis 5: In year 3, after two years of MSP implementation, we expect 
teachers in the treatment group (i.e., teachers targeted for MSP participation) to 
exhibit higher quality instruction, as measured by the alignment of instruction with 
state content standards, than teachers in the comparison group, after controlling for 
any year 1 differences. If teachers in the treatment group do, in fact, benefit from high 
quality MSP-supported professional development over time, their instruction should 
improve as a result (e.g., better alignment with standards and assessment or the utilization 
of higher-order instructional strategies in their classroom). 

Steps Completed in Study as oe Year 2 Progress Report 

At the end of the second year of the study (November 2004), the CCSSO/AIRAVCER 
team can report on six major steps carried out thus far: 

(1) Baseline survey with teachers (year 1). Surveys of Enacted Curriculum 
administered to teachers on site (Spring 2003) to establish baseline for instruction and 
professional development prior to MSP. The surveys covered instruction during the 
2002-03 school year, and professional development over the summer of 2002 and the 
2002-03 school years. Project staff traveled to 4 sites to administer teacher surveys to 
treatment and control groups of teachers, or provided training to local staff to administer 
surveys. In each local site, group administration to teachers was the preferred method of 
administration. A total of 389 surveys were completed by mathematics and science 
teachers (See Exhibit 3 for a summary of the surveys completed by MSP site and 
treatment condition). 

(2) On-site interviews and data collection (year 1). Interviews were conducted 
with MSP program coordinators and school district staff in spring 2003 to obtain core 
information about district standards, curricula, assessments and professional development 
activities supported by MSP. 

(3) Monthly data on professional development through the web-based 
activity log (years 1-2). Study staff designed a web-based tool, and teachers were asked 
to complete monthly log reports of all professional development activities completed 
over a 15-month period (July 2003 through September 2004). Steps were taken to 
provide an orientation for teachers on using the tool, and follow-up calls and mailings 
were made to encourage use. 

(4) Content coding of state standards and assessments for middle grade math 
and science for the states of the selected MSP sites (completed summer 2003). The 
coding results were used to analyze the degree of alignment between instruction and state 
standards — a measure of improvement. 

(5) MSP evaluation workshop for project directors, evaluators, and district staff 
for training on use of the Survey tools within their sites (fall, 2003), to provide a method 
of dissemination of evaluation tools tested in the project. All Cohort 1 MSP grantees 
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invited; with attendance by 50 participants supported by CCSSO RETA study grant and 
Utah State University RETA grant. 

(6) Initial report on analysis of baseline data from SEC Teacher Surveys and 
analysis of data from Professional Development Activity Log (PDAL). 

Section III of this report provides an analysis of variation in instructional practices and 
subject content taught across the study sample, using instructional scales and alignment 
content maps and indices, as well as an analysis of baseline professional development, 
drawing on SEC data collected in year 1 of the study. Section IV provides a preliminary 
analysis of variation in the professional development received by teachers during year 2 
of the study, based on data from the PDAL. 

Next Steps: Eollow-up surveys with treatment and comparison group math and science 
teachers will be conducted in fall 2004 and spring 2005 to measure change in instruction 
and professional development over time. 

Surveys of Enacted Curriculum: Instruments and Procedures 

CCSSO has worked with the AIR and WCER research team to develop a new, more in- 
depth approach to analyzing quality of professional development for teachers in math and 
science. One part of this approach has been to improve items on the Surveys of Enacted 
Curriculum (SEC) for collecting and reporting comparable and reliable data on detailed 
characteristics of many types of professional development activities experienced by math 
and science teachers. 

Since 1998, CCSSO has partnered with researchers and a collaborative of states to 
develop, test, and implement a system of survey tools — called Surveys of Enacted 
Curriculum — which has a primary function of reporting comparable data on key 
indicators of classroom instruction (both content and practices). The survey data 
collection and reporting system allows for detailed analysis of the quality of instruction 
as well as the quality of teacher preparation and development in their assigned subject 
area. 

The Surveys of Enacted Curriculum provide comprehensive data for analyzing teacher 
preparation, both quantitative course data and qualitative data on teacher views of their 
conditions for teaching and their own need for improvement. Three types of data are 
provided: Data on types and quality of current professional development, course 
preparation of teachers in their teaching field, and teacher beliefs about teaching and 
views about teaching conditions. 

To carry out the goals of the MSP longitudinal study, the research team revised and 
improved the items in the SEC tool planned for collecting data from teachers, based on 
research on the characteristics of high-quality professional development. These revisions 
addressed two main problems found in reviewing data recently reported in the DEC 
project. 
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First, the DEC survey had teachers answer questions about groups of professional 
development activities by subject/topic (e.g. implementing standards). While such items 
do provide an overall summary analysis of professional development, they do not provide 
sufficient information about how the specific activities are delivered or carried out, for 
example through one-day workshop, a mentoring process, or a summer course. Without 
detailed information, the survey responses impede analysis of the quality features of 
specific activities. 

Second, the items in the DEC project survey lacked a scale of responses from low to high 
for the quality criteria (e.g. content focus). The DEC items used yes/no responses, and 
there was an insufficient number of respond items to discern the degree to which the 
activity met a given criterion of quality. Teachers were only asked whether they felt the 
criterion was met. 

Survey Items for MSP study. For the MSP study, the research team developed a 
set of from 4 to 8 specific questions to assess teacher responses to each of the quality 
characteristics (i.e., coherence, active learning, content focus, and collective 
participation). Each item had a response scale constructed with a Likert-style format. 

For example, the items in the MSP survey version for science teachers to identify the 
degree of Active Learning in their professional development activities are the following: 

Exhibit 2: Items on Quality of Professional Development: Active Learning 
(New SEC Survey Items developed for MSP study) 



Thiukiog again about aO of your professional developnient aeTi\ illes in science or science edneadon 
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To review the complete revised Surveys of Enacted Curriculum section on Professional 
Development, see www.SECsurvev.org/tools . The review and revision process for items 
on quality of professional development was conducted in the MSP longitudinal study by 
the study project staff including researchers at CCSSO, AIR, and WCER, plus the 
expertise and assistance of the study advisory panel. 
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As a result of the PD items development and revision process under the MSP study, the 
total number of items addressing professional development in the Teacher Survey 
instrument doubled. The outcome was survey items that asked teachers to report on the 
characteristics of professional development activities which allowed for better analysis of 
quality. The format also was moved from a matrix to sets of items aimed toward each 
quality criterion. The responses by section are aggregated into summary scale measures. 

See Appendix C for graphic displays of data on professional development from the SEC 
baseline survey. 

SEC Survey Administration, Sample Size and Response Rate 

In spring 2003, teacher surveys were completed by a total of 214 mathematics and 185 science 
teachers in grades 6-12, across four MSP sites in three states. Five math and five science surveys 
were dropped because of partial survey completion or other missing information, yielding for 
analysis purposes a total sample of 389 teachers: 209 in mathematics and 180 in science (See 
Exhibit 3). Of these, 133 mathematics teachers and 88 science teachers are included in the 
treatment group. The comparison group is made up of 76 mathematics and 92 science teachers. 

Exhibit 3: Year 1 Sample Responses to Survey of Enacted Curriculum and Professional 
Development Activity Log (July 2003-September 2004): By MSP Sites and Treatment Status 



Intended 

Sample Year 1 SEC PDAL 

Response Completion 







Math 


Science 


Total 


Rate 


Math 


Science 


Total 


Rate 


MSP Site 




















Site 1 


91 


47 


31 


78 


86% 


37 


18 


55 


60% 


Site 2 


180 


78 


99 


177 


98% 


49 


63 


112 


62% 


Site 3 


93 


50 


28 


78 


84% 


34 


23 


57 


61% 


Site 4 


112 


34 


22 


56 


50% 


28 


21 


49 


44% 


Treatment Status 




















Comparison 


213 


76 


92 


168 


79% 


45 


54 


99 


46% 


Treatment 


263 


133 


88 


221 


84% 


103 


71 


174 


66% 


MSP_Site*Treatment Status 




















Site 1 -Comparison 


43 


19 


17 


36 


84% 


12 


8 


20 


47% 


Site 1 -Treatment 


48 


28 


14 


42 


88% 


25 


10 


35 


73% 


Site 2-Comparison 


68 


27 


40 


67 


99% 


14 


15 


29 


43% 


Site 2-Treatment 


112 


51 


59 


no 


98% 


35 


48 


83 


74% 


Site 3 -Comparison 


51 


15 


26 


41 


80% 


6 


22 


28 


55% 


Site 3-Treatment 


42 


35 


2 


37 


88% 


28 


1 


29 


69% 


Site 4-Comparison 


51 


15 


9 


24 


47% 


13 


9 


22 


43% 


Site 4-Treatment 


61 


19 


13 


32 


52% 


15 


12 


27 


44% 


Total 


476 


209 


180 


389 


82% 


148 


125 


273 


57% 



Note: ^ PDAL completion rate represents the number of teachers who completed usable PDAL data among the intended sample. 



Overview oe the PDAL Methods 

In order to examine the scope, nature, content, and quality of a wide array of professional 
development activities that teachers take part in over an extended period of time, AIR 
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developed the Professional Development Activity Log (PDAL), in collaboration with 
CCSSO and WCER. This new tool is built on AIR’s prior work on the National 
Evaluation of the Eisenhower Professional Development. The PDAL is a web-based, 
self-administered, longitudinal data collection tool with which teachers record their 
professional development experiences in detail with the assistance of a series of 
structured prompts. 

Teachers log on to their password-protected web account and fill out their PDAL at least 
once a month, even if they did not participate in any PD activities. In the PDAL, teachers 
are prompted each month to answer the following questions about each PD activity in 
which they participated: 

• Name of activity 

• Number of hours spent on each activity and its duration 

• Whether the activity is a one-time event or a continuous event (i.e., recurring over 
a number of months) 

• Type of activity (e.g., workshop, summer institute, study group) 

• Purpose of activity (e.g., strengthening subject matter knowledge) 

• PD quality features (e.g., active learning, coherence, collective participation; see 
Caret et al., 1999; Caret et al., 2001) 

• Content focus (e.g., algebraic concepts: absolute values, use of variables, etc.) 

• Instructional practice - instructional strategy topics covered in each activity (e.g., 
use of calculators, computers, or other educational technology) 

• Materials used during each activity 

Teachers may revisit the PDAL over multiple sessions as necessary to complete their 
monthly logs. They may also modify their entries until they elect to “finalize” the month. 
Some teachers may not have activities to report for some of the months during the study 
period. These teachers can report their “inactive” PD status for the month by simply 
clicking on a button, which states, “I did not participate in any formal professional 
development activities this month.” 

PDAL Respondents 

A total of 273 math and science teachers from 4 MSP sites have participated in the 
PDAL. Overall, 57% of the teachers in the sample completed the PDAL for at least one 
month. However, as Exhibit 3 shows, the rates vary by MSP sites and MSP program 
participation status. For example, sites 1, 2, and 3 achieved about the same rates: 60%, 
62%, and 61%, respectively. But site 4 lagged behind, reaching only 44%. Treatment 
teachers were more likely to complete the PDAL compared with their non-treatment 
counterparts: 66% vs. 46%. For the purpose of the analysis for this report, we used the 
PDAL data that were completed by September 15, 2003. The data set includes logs 
covering the period from July 2003 through August, 2004. By September 15, 2004, 1,789 
monthly logs have been completed by 273 teachers. 
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III. Surveys of Enacted Curriculum: Baseline Teacher Survey 

Surveys were initially designed by CCSSO and WCER through prior studies (Blank, et al, 2001; 
Porter, 2002; Blank, 2004). The version for the present study was developed by the study team 
with assistance from the study advisory panel (See www.SECsurvev.org/ Tools). Teacher 
surveys were administered in spring 2003 with the assistance of local site coordinators. Training 
for administration of the surveys was provided project staff in each of the four sites. In most 
cases, administration of the surveys occurred during faculty meetings held specifically for survey 
completion. In some cases teachers were allowed to complete the surveys at their convenience, 
and either centrally collected and sent to WCER in bulk, or in a few cases sent individually be 
teachers. The highest response rates were noted in those places were group administration of the 
survey was conducted. The response rate across the four sites, based on our best estimates of 
anticipated program participation was 84%. [Appendix Table A for summary data.] 

Surveys were received completed by a total of 214 mathematics and 185 science teachers in 
grades 6-12 across four MSP sites in three states. Eor analysis purposes, the following sample 
size with complete data was obtained: 





Math 


Science 


Complete 


209 


180 


Treatment 


133 


88 


Comparison 


76 


92 



SEC Data Overview 

The SEC instruments collect a broad range of information about the educational context. The 
survey is organized into 10 sections that provide a good sense of the range of data collected: 

School & Class Description Instructional Readiness 

Use of Homework Teacher Opinions 

Instructional Activities Professional Development Activities 

Instructional Influences Teacher Characteristics 

Use of Assessments Instructional Content 

Survey results are typically reported in profiles and scales. Profiles highlight results of 
individual items grouped by survey section. Scales report summary measures that aggregate 
individual items based on some common construct. Because of the sheer size and breadth of the 
data set, a variety of scale measures can and have been constructed. For information on the full 
set of scales employed in the initial analyses, the items they are constructed from, and measures 
of scale reliability, see Appendix B. Data from the Instructional Content section of the Survey 
are reported in two dimensional content maps or graphs — topics by expectations for learning (or 
cognitive demand). For the MSP project, the Instructional Activities scales were redesigned to 
parallel cognitive demand dimension of the content survey. 
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The following scales and summary measures are used in the initial analyses reported here: 

Scales related to instructional practice 

Assessment Use 
Climate of Trust 

Communicate Understanding (science) 

Analyze Information 
Active Learning 

Scales related to professional development (PD) activities 



Influence of Standards on Instruction 
Perform Procedures 
Demonstrate Understanding (math) 
Make Connections 



Frequency of PD Activities 
Active PD 

PD with collective participation 
PD with focus on student data 
PD with focus on student learning 



PD Hours 
Coherent PD 

PD with focus on subject matter content 
PD with focus on standards & instruction 



Summary Measures related to Instructional Content 



Topic Coverage Categories of Cognitive Demand 

Alignment to Tests Alignment to Standards 

In addition, we considered the following teacher characteristics in our initial analyses: teacher 
gender and ethnicity, class organization, class size, course type, teacher perception of student 
ability, teacher experience, and number of years teaching at this school. 

In the analyses conducted for this report we ask two primary questions: 

1) Do our treatment and comparison groups differ significantly on important 
characteristics (Hypothesis 2)? 

2) What are the pre-existing relationships among key variables in the baseline data 
results? Also, where relevant, we note significant differences across the four participating sites 
(Hypothesis 1). While sites are likely to vary on some characteristics (and important to note), we 
anticipate that comparisons between the treatment and comparison groups will ideally yield little 
or no significant differences between the two groups on key variables related to MSP program 
treatments. 



To answer these questions, analyses of variance and correlations were examined in concert with 
descriptive results for the variables identified above. Results of those analyses are reported here. 
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Teacher Characteristics 

The SEC instruments collect data on the following teacher characteristics: 

Gender 

Ethnicity 

# Years teaching science/math 

# Years at current school 
Highest degree held 

Type of teaching certification(s) held 

These variables were examined for significant differences between the treatment and comparison 
groups of teachers. 
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No significant differences were found between the treatment and comparison groups of 
mathematics teachers on the basis of gender, ethnicity, highest degree held, or certifications held. 
However, among responding science teachers we do note a higher proportion of female teachers 
in the treatment group (70%) than the comparison group (60%). The sample also indicates a 
significantly higher proportion of 
Hispanic or Latino teachers in the 
comparison group (21%) compared to 
the treatment group (9%). However, 
when mathematics and science teachers 
were grouped together, no significant 
differences were found between 
treatment and comparison groups in 
terms of either gender or ethnicity. 

As indicated in Exhibit 4 (right column), 
mathematics teachers in the comparison 
group report more years experience at 
teaching mathematics than those in the 
treatment group. Among science 
teachers no significant differences are 
noted between the comparison and 
treatment groups in terms of years of 
experience, or years in their current 
school (see Exhibit 4, left column). 

Instructional Activities 
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The SEC data set provides five scale measures associated with instructional activities related to 
mathematics and science instruction. Though the instructional practice scales parallel one 
another across subject areas, it should be noted that the scales differ by subject both in terms of 
the specific survey items that define the scales, as well as (in some cases) slight difference in the 
conceptual constructs being measured. Listed by subject, these scale measures are: 
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Mathematics 
Perform Procedures 
Demonstrate Understanding 
Conjecture, Hypothesize, Prove 
Make Connections 
Active Learning 



Science 

Perform Procedures 
Communicate Understanding 
Analyze Information 
Make Connections 
Active Learning 



(See Appendix B for scale items/definitions and results of reliability analyses.) 



No significant differences were found between the comparison and treatment groups on four of 
the five scale measures related to instructional activities (see Exhibit 5) and all six variables 
related to science instruction (see Exhibit 3), and these results indicate, in general teachers 
selected for MSP participation appeared to be similar in their instruction to comparison teachers. 
Only the Active Learning Scale for Mathematics indicates any significant difference among 
treatment and comparison teacher groups. Analyses of variance also indicate no statistically 
significant differences among sites on these measures, indicating that instructional approaches 
appeared similar across our sites. 

Note that the Active Learning scale shares some items with other scale measures (see Appendix 
B). These scales are being employed in an exploratory manner at this point, and only scales 
consisting of mutually exclusive items will be utilized in the formal analysis of longitudinal 
results. 



Four of the above measures parallel measures of cognitive demand collected in the instructional 
content section of the survey, but are based on items from the instructional practices section of 
the SEC surveys. While only “Making Connections” shows a significant and positive correlation 
with its cognitive demand counterpart (r = .22; p = .003), the general pattern of relative emphasis 
across the two sets of student expectation measures are similar (compare Exhibits 5 & 6 with 
Exhibits 7 & 8). While it might seem reasonable to expect higher correlations between similar 
measures calculated from the practice and the content sections of the survey, it should be noted 
that the content measures are orthogonal, and account for 100% or instructional time, while the 
measures based on instructional practice are not orthogonal and do not account for 100% of 
instructional time. We believe this measurement difference contributes to the low correlations 
for similar measures. 



Instructional Content 

In the SEC, instructional content is described in terms of subject -specific topic coverage and 
cognitive demand. K-8 mathematics uses a list of 103 mathematics topics, while the science 
includes 149 topics. Both subjects use five categories of cognitive demand (see above), though 
the specific description for each category varies by subject. Respondents are asked to report first 
on the total amount of instructional time (in class periods) spent on the various topics listed. 
Respondents then report the relative emphasis across each category of cognitive demand for 
every topic that receives some instructional time. Using this approach to content description 
allows investigators to examine results based on the intersection of topic and cognitive demand, 
as well as examination of each dimension (topic, cognitive demand) separately. 
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Summary results of instructional content are presented in two ways. Content summary measures 
report results separately for each of the two dimensions used to describe subject matter content 
(topics and cognitive demand). Content maps show the intersection of the two dimensions using 
a topographic or surface area map. Content maps of instruction, assessments, and standards are 
provided in Appendix G. Discussion here will be limited to marginal results for instructional 
content. 
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Cognitive Demand 

Exhibits 7 & 8 display teacher reports of their emphases on the five cognitive demands. The 
pattern of emphases across categories of cognitive demand varies noticeably by subject. 
Mathematics teachers consistently report somewhat more emphasis on performing procedures 
and demonstrating understanding than the other categories. This pattern persists whether the 
sample is disaggregated by site or by treatment vs. comparison teachers. The only significant 
mean difference is noted for Conjecture, Generalize & Prove in Site 1, where somewhat less 
emphasis is reported for this category. 

In science we note differences in the placement of relative emphases by sites, but none rising to 
the level of significance. When disaggregated by treatment vs. comparison, the results show that 
the two groups of teachers in the sample had similar averages and variation at the beginning of 
the study. 
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Topic Coverage 

Because the list of topics used to describe content differs between K-8 and high school grade 
levels, we conducted separate analyses of differences in topics covered by middle school and 
high school teachers. Since high school teachers were not the original focus of this study, and 
only two sites included high school teachers in the sample, results are presented here for middle 
school grades only. For K-8 mathematics instruction only one content area (Geometric 
Concepts) showed significant mean differences between comparison and treatment groups, with 
the comparison group reporting slightly less time spent on geometric concepts (12%) than 
reported for the treatment group (15%). In general, the data indicate consistency in mathematics 
topic coverage across treatment and comparison teachers in the sample, and across sites. Site 1 
provides one exception to this pattern; comparison of mathematics topic coverage across sites 
reveals significantly greater variation among Site 1 mathematics teachers with respect to the 
amount of time spent using instructional technology. This is the only significant difference in 
topic coverage noted between sites, particularly insofar as Site 1 has increasing the amount of 
time using of instructional technology among mathematics teachers as a primary goal of their 
program initiative (see Fig. 6). For this reason it should also be noted that treatment teachers in 
site 1 report slightly more time spent on instructional technology (mean = 0.04) than do 
comparison teachers (mean = 0.03). 
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Because the topic list for science is much larger than mathematics, and organized into 25 content 
areas instead of 8, Exhibit 10 lists only those content areas for which significant mean 
differences were noted either among sites or between treatment and comparison groups. Of the 
content areas listed in Exhibit 10, only ‘Energy’ does not indicate significant variation between 
sites. Energy is however a content area where the mean difference between comparison and 
treatment groups is statistically significant, with the treatment group reporting more time on 
energy topics than the comparison group. Site 1 is particularly notable for the wide variation 
among teacher reports of time spent on topics related to Measurement & Calculation in Science 
and especially topics related to Science & Technology. 
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Alignment Analyses 

Alignment in these analyses is a quantitative measure used to describe the extent of similarity 
between instruction, assessments, and standards. Our alignment indicator has a range of 0 to 1, 
where 0 represents complete lack of alignment and 1 represents perfect alignment. For example, 
alignment measure of 1.00 between instruction and some assessment would indicate that only 
content represented on the assessment was taught, and that the relative proportion of time spent 
in instruction on any given topic at any given level of cognitive demand mirrored the relative 
proportion of score points on the assessment associated with that topic and category of cognitive 
demand. Similarly, an alignment measure of 0.00 between an instruction and some assessment 
would indicate that no topic assessed was included as part of instruction. 
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During year 1 of the project, mathematics and science content standards for each of the three 
states represented in the sample were content analyzed using the two-dimensional descriptive 
language employed for describing subject-matter content. State assessments were also content 
analyzed where access was available Exhibit 1 1 lists the standards and assessments that have 
been content analyzed for the three states represented in the sample. Note that while our study 
was originally designed to focus on middle school, two of the four sites participating in the study 
have programs that target both middle school and high school teachers. As a result, the sample 
includes high school teachers in sites 1 & 2. High school assessments and standards for State 2 
only have been content analyzed at this point. To the extent possible, content analyses will be 
conducted in the upcoming year on standards and assessments missing in Exhibit 1 1, in order to 
provide relevant targets for measuring instructional alignment. 



Exhibit 1 1 





Mathematics 


Science 




Assessment 


Standards 


Assessment 


Standards 


State 1 


Gr. 8 


Gr. 5-8 


- 


Gr. 5-8 


State 1 (H.S.) 




- 


- 


- 


State 2 


-- 


Gr. 5-7 


Gr. 7 


Gr. 7 


State 2 (H.S. 


Gr. 9 


Gr. 9 


Gr. 9 


Gr. 9 


State 3 


Gr. 6 & 8 


Gr. 6 & 8 


i 


Gr. 8 



Where possible, an alignment index was calculated between instruction and the most relevant 
standard and/or assessment. Where possible, instructional alignment to a standard or assessment 
was targeted to the specific grade for which instruction was reported on. Thus in state 3, sixth 
grade mathematics teachers had alignment calculated on that state’s grade 6 assessment and 
standards. Where the specific grade of instruction reported on was not available, the next closest 
grade was selected as the alignment target. Eor example grade 7 mathematics teachers in state 3 
had alignment targeted to the grade 8 standards and assessment, while all science teachers in 
state 3 had their instruction aligned to the grade 8 science standards, since this was the only 
relevant target available. Because the content language differs between grades K-8 and high 
school, all middle school mathematics teachers in state 2 had their alignment index based on the 
grade band 5-7 mathematics standards for state 2. 

Alisnment of Instruction to Assessments & Standards 

Using these parameters, alignment indices were calculated for targeted standards and 
assessments (where available). Across the sample of mathematics teachers alignment to the 
assessment ranged from 0.1 1 to 0.34, (mean = 0.21; std. dev. = 0.04), compared to a range of 
0.05 to 0.30 (mn. = 0.17, and std dev. = 0.05) for standards. 

As can be seen in Exhibit 12, the pattern of higher alignment of instruction to assessments than 
for instruction to standards persists across sites. Even in Site 1, with its substantially higher 
alignment to standards compared to the other sites, teachers’ instruction (content coverage?) 
appears to be more highly aligned to assessments than to standards. While the treatment group 
shows slightly higher alignment of instruction to both standards and assessments, compared to 
the comparison group, these mean differences were not found to be significant upon analysis. 
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The alignment of teachers’ instruction to both standards and assessments in Site Iwere, however, 
significantly higher than in the other three sites. It is worth noting that the standards for State 1 
(in which Site 1 is located) are set at the intermediate level grade band (grades 5-8), and 
represent the broadest range of target grades among the standards represented. State 2 (Site 2), 
with the next broadest target range (grades 5-7) for mathematics standards also had the next 
highest mean level across the sites, though not by a statistically significant amount. State 3 
(Sites 3 & 4) in turn had the most specific targets, with grade 6 and grade 8 standards analyzed, 
and yielded the lowest mean level of alignment across the four sites. Whether this pattern of 
better alignment to more broadly defined targets over grade-specific targets will maintain as 
more documents and more descriptions of instruction become analyzed is not known. However 
such a pattern would be consistent with the perhaps too simplistic notion that a broader target is 
easier to hit. 

Exhibit 13 reports alignment results for science. Across the four sites alignment to assessments 
ranged from 0.03 to 0.21 (mean = 0.13; std.dev. = 0.04), and 0.00 to 0.30 (mean = 0.15; std.dev. 
= 0.05) for standards. Science instruction reported by teachers in the sample appears to be 
somewhat better aligned to standards than to assessments. Though standards and assessments 
from more states would need to be analyzed and compared before we could say this pattern is 
persistent, it is a pattern consistent with the argument that science assessments are new to high 
stakes assessments, and are less visible to science teachers than are science standards. The site- 
by-site patterns are also interesting to consider. Alignment to standards for site 2 shows 
dramatically greater variation than seen in the other three sites, as well as the highest mean 
alignment measure across sites. 
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Alignment of Assessments to Standards 

Most discussions about alignment concern alignment between standards and assessments. A key 
element of the assessment-based accountability component of the standards-based reform is the 
requirement that assessments be appropriately aligned to standards. While we have no theory- 
based criteria for what represents good alignment between standards and assessments using our 
quantitative approach to alignment, we do have a growing body of empirical results that give a 
sense of what currently represents typical alignment. For mathematics, alignment between state 
tests and standards typically runs between 0.31 and 0.47 (Porter, 2002). 

Exhibit 14 reports alignment results between standards and assessments. The main diagonal of 
Exhibit 14 reports results for alignment between a given state’s content standards and that same 
state’s assessment, where we had a grade level match between standard and assessment. The 
off-diagonal measures report alignment for standards and same-grade assessments from other 
states. Assuming that recent efforts to align standards and assessments within states have been 
successful, we would expect to see the average alignment for a state’s assessment to its own 
standards to be higher than the average alignment of state assessments to other state standards. 
The average for the main diagonal in Exhibit 14 is 0.40, while the off-diagonal alignment 
average is 0.29, supporting the expectation that state assessments will be more highly aligned to 
their own state content standards than to standards from other states. 



Exhibit 14 

Alignment of State Mathematics Assessments to Standards 





Assessments 


Standards 


State 1 Gr.8 


State 2 Gr.9 


State 3 Gr.6 


State 3 Gr.8 


State 1 Gr.8 


0.38 




0.24 


0.27 


State 2 Gr.9 




0.34 






State 3 Gr.6 


0.36 




0.48 




State 3 Gr.8 


0.30 






0.38 



Exhibit 15 reports similar results for science. Note that no science assessments were available 
for analysis from State 1, and only one grade specific match between state standard and 
assessment was available for comparison. Due to the limited sample of science documents 
analyzed, particularly in matching grade-specific standards to same grade assessments, similar 
analyses are not possible at this time for science. With what data is available for looking at 
alignment between science standards and assessments, the pattern appears to suggest generally 
lower alignment in science than in mathematics. If this pattern persists with more analyses, we 
would expect that this is consistent with states only recently beginning to put attention on high- 
stakes science assessments and resulting interest in alignment between assessments and 
standards, whereas these issues have been a focus in mathematics for some years now. 
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Exhibit 1 5 

Alignment of State Science Assessments to Standards 





Assessments 


Standards 


State 1 
Gr.8 


State 2 
Gr.7 


State 3 
Gr.5 


State 1 Gr.8 


-- 


0.14 


0.23 


State 2 Gr.7 


-- 


0.15 


0.17 


State 3 Gr.8 


-- 


0.15 


0.15 



Re lationships among key variables 

In addition to simple descriptive statistics and analyses of variance, correlations between key 
variables (listed above, see SEC Data Overview) were examined to identify relationships 
existing at the baseline. In particular, relationships between professional development activities 
and instructional practice were examined for pre-existing (prior to treatment) relationships. 

Among both mathematics and science teachers, several PD-related scales were significantly 
correlated with amount of assessment use and influence of standards are noted (see Exhibit 16). 



Exhibit 16 

Correlations between Classroom Assessment Use & Features of PD 





Mathematics 


Science 




r Value 


p Value 


r Value 


p Value 


PD Frequency 


0.31 


0.000 


0.42 


0.000 


PD Hours 


0.26 


0.000 


0.35 


0.000 


Active PD 


0.41 


0.000 


0.44 


0.000 


Coherent PD 


- 




0.43 


0.000 


Collective PD 


0.18 


0.010 


0.22 


0.004 


PD w/ Content Focus 


0.26 


0.000 


0.40 


0.000 


PD w/ Data Focus 


0.32 


0.000 


0.46 


0.000 


PD w/ Stndrds/lnstr. Focus 


0.24 


0.000 


0.42 


0.000 


PD w/ Student Lrng. Focus 


0.32 


0.000 


0.49 


0.000 



While the results of Exhibit 16 indicate a strong relationship between the use of assessments in 
the classroom and participation in professional development activities, the causal relationship 
among these variables remains unclear. These data represent pre-existing, baseline relationships, 
and no evidence is available to determine whether PD fosters classroom assessment use, or vice 
versa. Examining these relationships for patterns of change after the second round of data 
collection, particularly among teachers in the treatment group, and with additional information 
on the nature of the professional development activities engaged in by teachers, should help to 
discern effects on these relationships of participation in MSP sponsored professional 
development activities. 
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Exhibit 1 7 

Correlations between Influence of Standards on Practice & Features of PD 





Mathematics 


Science 




r Value 


p Value 


r Value 


p Value 


PD Frequency 


0.26 


0.000 


0.22 


0.003 


PD Hours 


0.25 


0.000 


0.16 


0.042 


Active PD 


0.26 


0.000 


0.30 


0.000 


Coherent PD 


0.24 


0.001 


0.50 


0.000 


Collective PD 


0.42 


0.000 


0.33 


0.000 


PD w/ Content Focus 


0.64 


\ 0.000 


0.63 


0.000 


PD w/ Data Focus 


0.53 


\ 0.000 


0.47 


0.000 


PD w/ Stndrds./lnstr. Focus 


0.67 


! 0.000 


0.63 


\ 0.000 


PD w/ Student Lrng. Focus 


0.37 


\ 0.000 


0.46 


\ 0.000 



Similar relationships between professional development participation and the level of influence 
of standards on classroom practices can be seen in Exhibit 17, with significant and positive 
correlations reported for all PD-related scales. Again the nature of these relationships, especially 
with respect to cause and effect cannot be determined from these data. 

While little can be said about the causal nature of the relationship between participation in 
professional development activities and either use of assessments in the classroom or the degree 
of influence of standards on classroom practice, the strong and persistent correlations between 
these variables do suggest the utility of these scales for looking at relationships between practice 
and professional development. 

The strong relationships to participation in professional development activities noted above do 
not persist when looking at more specific classroom practices. For example, teachers’ emphasis 
on active learning for their students is associated with three of the nine PD-related variables in 
mathematics, and only two for science (see Exhibit 18). Curiously, in mathematics the 
frequency of professional development participation is positively correlated to active learning, 
but not the amount of PD (PD hours). The reverse is true for science. Both subjects however 
show a positive correlation between active learning and professional development activities that 
engage teachers in active learning, as would be expected. Collective participation in professional 
development activities also shows a positive relationship with active learning in mathematics, 
but not science. 



Exhibit 1 8 



Correlations between Active Learning & Features of PD 



Correlation between 


Mathematics 


Science 


Active Learning & 


r Value 


p Value 


r Value 


p Value 


PD Frequency 


0.19 


0.007 


-- 




PD Hours 


-- 




0.18 


0.022 


Active PD 


0.26 


0.000 


0.20 


0.007 


Coherent PD 


-- 




-- 




Collective PD 


0.16 


0.022 






PD w/ Content Focus 


-- 








PD w/ Data Focus 


“ 


1 






PD w/ Stndrds/lnstr. Focus 


-- 




-- 




PD w/ Student Lrng. Focus 


- 




-- 
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Recall that two alignment variables were calculated based on teacher reports of instructional 
content and content analyses of relevant state standards and assessments. Correlation results for 
alignment to standards are reported in Exhibit 19. Only one PD-related variable appears related 
to alignment to standards for mathematics teachers, frequency of PD participation. However, a 
number of these variables are associated with alignment to standards among science teachers. In 
science at least, as alignment of instruction to standards goes up, so does the amount of 
participation in professional development reported. Additionally, several of the quality 
indicators of professional development appear to be associated with increased alignment to 
standards. Once again, causal statements are not warranted at this time, however we expect that 
in mathematics gains in alignment will be associated with participation in MSP-sponsored 
professional development activities. 



Exhibit 1 9 

Correlations between Alignment to Standards & Features of PD 



Correlation between 


Mathematics 


Science 


Alignment to Standards & 


r Value 


p Value 


r Value 


p Value 


PD Frequency 


0.15 


0.050 


0.30 


0.000 


PD Hours 


- 




0.17 


0.037 


Active PD 






0.19 


0.015 


Coherent PD 


- 




0.20 


0.014 


Collective PD 


- 




- 




PD w/ Content Focus 




i 


0.20 


0.011 


PD w/ Data Focus 






- 




PD w/ Stndrds/lnstr. Focus 


- 




0.19 


0.015 


PD w/ Student Lrng. Focus 


" 




0.18 


0.025 



Conclusions for SEC Year 1 Data Analyses 

As previously stated, the primary focus of year 1 SEC data analyses was to compare the 
characteristics of treatment and comparison teachers, and to determine any similarities and 
differences in the baseline data across sites. 

With respect to treatment and comparison group characteristics, we note that mathematics 
teachers in the comparison group have somewhat more years experience on average than 
reported by mathematics teachers in the treatment group. Females are represented more among 
science teachers in the treatment group (70%) than the comparison group (60%). Latinos are 
more heavily represented among comparison group science teachers (21%), than their treatment 
group counterparts (9%). No other differences in teacher characteristics are noted for the 
treatment and comparison groups. 

Only one scale measure related to student activities (Active Learning) showed significant 
differences between treatment and control groups, and only for mathematics, with treatment 
mathematics teachers reporting more use of active learning in the classroom than reported by 
comparison mathematics teachers. No significant differences in instructional activities were 
noted between treatment and comparison groups in science. 
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Teacher reports of instructional content indicate that treatment mathematics teachers spend 
slightly more time on Geometric Concepts than do comparison group mathematics teachers. No 
significant differences were noted on the content dimension of cognitive demand between 
treatment and comparison groups for either subject. While differences in content coverage were 
noted for both sites and treatment/comparison groups, because science typically shows great 
diversity of topic coverage among teachers, these results are not unexpected. No significant 
difference in mathematics topic coverage is noted between treatment and comparison groups. 
Similarly, no significant differences between groups are noted for either alignment measure. 

Based on these analyses it is believed that the comparison and treatment groups are sufficiently 
similar to be useful for detecting differences in outcomes attributable to participation in MSP 
sponsored professional development activities. Additionally, a number of pre-existing 
relationships were noted among key variables (described above). These pre-existing 
relationships will be taken into consideration, and to the extent necessary controlled for, in 
conducting the longitudinal analyses to follow Year 3 data collection. 
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IV. Professional Development Activity Log Results 

In this section, we first describe how the PDAL was administered for this study, while 
underscoring the nature of the PDAL data and its analytic potential and advantages. 

Then, we explain how professional development scales were constructed and present 
basic descriptive statistics for the constructed scales. Lastly, we present the results of our 
preliminary data analysis that was undertaken to test hypotheses 3 and 4 for the study. 

Administration of PDAL 

The PDAL data collection was launched in July, 2003 and ended on November 15, 2004. 
Teachers who had completed the SEC year 1 and/or who taught in schools in the sample 
received an introduction packet in the mail. The packets included a letter introducing 
them to the PDAL, instructions on how to sign-on to and complete the PDAL, and a 
glossary of terms (e.g., definition of institute). Further, each teacher was provided with 
an email and toll-free phone number to call when they needed help with the PDAL. 

Teachers were instructed to begin creating log entries for July, 2003. Subsequent month 
logs would be automatically generated for teachers to fill out starting from the first day of 
each month. Teachers were asked to complete reporting on a month by the 15* of the 
following month. Teachers created separate logs for each professional development 
activity they participated in each month. Teachers could revisit the PDAL (over multiple 
sessions) as necessary to complete their monthly activity logs. They could also modify 
their log entries until they chose to “finalize” the month. If the activity continued beyond 
the first month, they would continue to report on that same activity in following months. 
Teachers who did not have any professional development activities in a given month 
were asked to indicate their inactive status for the month by clicking on a specific link on 
their PDAL website, which states, “I did not participate in any formal professional 
development activities this month.” Regardless of their active or inactive status for the 
month, if teachers had participated in any informal activities, they were asked to report 
how many hours they engaged in informal self-directed learning activities in the month 
and if they used any of the informal self-directed learning in their classroom.’ 

We used several strategies to encourage teachers to respond to the PDAL. First, reminder 
emails were sent each month. At first, the emails simply reminded the users to fill out 
their PDAL. Over the course of the project, the emails included a list of all the activity 
logs the teachers had created for each month. This list also provided explicit instructions 
on what the teachers still had to do in order to complete their PDAL for a given month. 
Additionally, postcards were sent intermittently to teachers throughout the study (e.g., as 
a reminder at the beginning of the school year). Next, a subcontractor was hired to make 



* Informal professional development was defined in the PDAL instructions as an independent, self-directed 
learning opportunity that teachers select themselves, on the basis of their personal or professional interests. 
Examples are using Internet sites to plan lessons or do research; reading a specific journal to learn of the 
latest research in their field; and meeting their fellow teachers in informal ways to expand their interests 
and knowledge. 
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2 

monthly reminder phone calls to all in-scope teachers. The subcontractor reminded the 
teachers to complete their current and outstanding monthly PDAL logs, provided 
technical assistance, recorded inactive months, and updated contact information. We 
made an incentive payment of $50 to the PDAL participants when they filled out the first 
month’s log and another payment of $50 when they completed and finalized all their 15 
months of the PDAL. 

Advantages of the PDAL 

The PDAL has a number of advantages over existing tools (Yoon et al., 2004). First, 
monthly logs are likely to provide accurate, time- sensitive information about teachers’ 

PD experiences. By design, the PDAL allows teachers to enter real-time data as their PD 
activities occur. At the minimum, it collects information on teachers’ PD activities in the 
immediate past month(s) and helps reduce a recall bias from retrospective data. Second, 
with the PDAL, we can avoid problems with data aggregation when teachers report their 
PD experiences that occurred over a period of time. Researchers can aggregate basic data 
to selected level(s) of aggregation for analysis (e.g., activity-level or teacher-level). 
Survey methods that ask teachers about their overall and cumulative PD experience 
cannot capture detailed activity-specific information. Third, because we take an inclusive 
approach to PD activities (i.e., not limited to MSP-sponsored activities), we can examine 
teachers’ PD experiences in a larger context. Fourth, we emphasize behavioral indicators 
of teachers’ PD experiences (e.g., the frequency and contact time; opportunities for active 
learning; collective participation). Fifth, by taking advantage of structured prompts for 
skip patterns, the PDAL generates context-sensitive questions. For example, if teachers 
indicate they did not cover certain content areas, the PDAL skips questions regarding 
those areas. This feature of the PDAL alleviates teachers’ burden. Lastly, because 
teachers’ log entries are automatically saved in a database as they respond, we reduce the 
chance of data entry errors. In sum, with the PDAL, we expect to obtain more valid and 
reliable data on teachers’ specific PD experiences. 

Patterns of Participation 

The advantages of the PDAL come in large part from the nature of the data that it 
produces; the PDAL captures the complexity of teachers’ participation in a wide variety 
of professional development activities. 

The pattern of participation in professional development (PD) may vary widely by 
individual teachers, by the month of the year, or by the type of activities. To illustrate a 
variety of patterns of PD participation, we created hypothetical data. Exhibit 20 displays 
patterns of participation in 7 PD activities (A through G) by 5 teachers over the period of 
8 months (July 2003 through February 2004). The total number of hypothetical monthly 
logs constructed for the combination of the 7 PD activities and 5 teachers is 20. Each “1” 
in the cell of Exhibit 20 represents a monthly log. First, some teachers may be more 
active in PD than others. As hypothetical data in Exhibit 20 illustrates, for example, Mr. 



^ Those who retired, moved out of school district, or taught a subject other than mathematics or science 
were treated as out-of-scope of the PDAL data collection. 
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Anderson participated in three activities (A, B, and C) during the 8-month period and 
completed 6 separate monthly logs for the activities. During the same period, however, 
Mrs. Smith kept only one monthly log for a single activity (G). 

Second, the activity level may fluctuate by the month of the year. For example, four of 
five teachers were actively engaged in PD in August 2003, while only one teacher was 
active in PD in December 2003. 

Exhibit 20: Basic monthly activity log data: Log-level data disaggregated by teacher by 

activity by month (hypothetical data) 







Jul- 


Aug- 


Sep- 


Oct- 


Nov- 


Dec- 


Jan- Eeb- 


#of 


Teacher 


Activity 


03 


03 


03 


03 


03 


03 


04 04 


logs 


Mr. Anderson 


A 


1 


1 












2 




B 






1 




1 




1 


3 




C 








1 








1 


Ms. Lopez 


A 


1 


1 












2 




D 










1 


1 


1 1 


4 


Mrs. Kelly 


E 




1 


1 










2 


Mr. Lee 


E 


1 


1 


1 










3 




E 






1 


1 








2 


Mrs. Smith 


G 


1 


1 


# of logs per month 


3 


4 


4 


2 


3 


1 


2 1 


20 



Third, the pattern of participation in PD may be determined by the type of activity being 
offered. For instance, activities C and G are one-time events that occur within a single 
month. A typical district workshop and a national conference are examples of such one- 
time activities. Other activities like A, B, and D continue into following months. A study 
group or task force may be such continuous activities that require a teacher’s prolonged 
involvement. Some continuous activities such as A and D are conducted in consecutive 
months, while others like B in non-consecutive months (e.g., every other month). Further, 
some teachers may attend multiple activities within a month. For example, Mr. Lee took 
part in activities E and F during September 2003. Still others may not have any activity to 
report for some of the months during the study period. For example, Mr. Anderson was 
inactive during October 2003 and February 2004. 

Lastly, some activities (e.g., A and E) are commonly reported by more than one teacher, 
while others (e.g., B, C, and G) are reported by only a single teacher. For example, Mr. 
Anderson and Ms. Lopez filled out separate logs for activity A in which they shared 
common experience during July and August of 2003. Even for the common activity, two 
teachers may differ in their overall assessment of their PD experience. In sum, the PDAL 
exposes these various patterns of activity participation through collecting disaggregate 
monthly log data about each and every professional development activity (or lack 
thereof) that individual teachers experience over the course of the study period. 
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Levels of the PDAL Data 

As Exhibit 20 illustrates, a monthly log is the basic level (or unit) of raw data collected 
with the PDAL; that is, a record for an activity for a month. And it is a basic building 
block of a complex PDAL data structure. For instance, Mr. Anderson participated in three 
activities (A, B, and C) over a period of 8 months and completed 6 separate monthly logs: 
2 for A, 3 for B, and 1 for C. More specifically, activity A may represent a summer 
institute that extended over a couple of month; activity B, which recurred in every other 
month over a span of 5 months, may be a mentoring activity; and activity C may reflect a 
district-sponsored workshop. Since the PDAL requests teachers to report about their PD 
on a monthly basis, we anticipate some variation in the quantity and quality of PD 
between the monthly logs that were created for the same activities. For example, between 
the two monthly logs submitted by Mr. Anderson for activity A, he may report different 
amount of contact hours and different amount of content focus depending on the month 
of reporting. In sum, as a disaggregate unit of observation, monthly logs for each activity 
will allow us to document in detail a great number of possibilities of PD activities with 
varying quantities and qualities that teachers experience over an extended period of time. 

Using disaggregate, monthly log level data as basic building blocks, we can construct an 
array of new aggregate data that are useful to describe various PD activity profiles for 
teachers. First, we can produce teacher-activity level data by aggregating the basic 
monthly log level data across months for each activity. As the last column of Exhibit 20 
displays, there are 9 sets of teacher-activity level data. They represent unique 
combinations between teachers and activities. For example, the first set of teacher- 
activity level data (as shown in the first row of Exhibit 20) represents Mr. Anderson’s 
participation is Activity A, which can be produced by aggregating two monthly log 
records for July and August 2003. This set of teacher-activity combination is 
distinguished from another set of teacher-activity combination between Ms. Lopez and 
Activity A (as shown in the fourth row of Exhibit 20). 

Hence, Mr. Anderson’s PD profile can be summarized with three sets of teacher-activity 
level data, which represent his participation in three separate activities. Since the quality 
of one PD activity may differ from that of another, it is important to create separate 
measures of PD quality such as content focus or active learning for each activity. In the 
case of a one-time activity (e.g., activity C) which is bound to a month, there is no need 
to aggregate data across months. But, in the case of continuous activities such as A and 
B, we need to combine a series of monthly log records that Mr. Anderson filled out for 
the activities. For example, to obtain Mr. Anderson’s total contact hours for activity A, 
we need to sum up the contact hours reported for July and August 2003. In addition, we 
can compute mean contact hours for the same activity. In a similar manner, we can 
compute other aggregate measures of PD quality such as coherence for each activity by 
averaging out corresponding quality indices across months. In this report, we introduce 
teacher-activity level data as one way of data aggregation and underscore its potential 
usefulness in describing various PD activity profiles for teachers. 
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Second, we can produce activity level data by aggregating the basic monthly log level 
data across months and teachers. As the last column of Exhibit 21 shows, a total of 20 
monthly log records were aggregated to produce 7 different PD activity profiles (or 7 sets 
of activity level data). For example, 4 monthly logs created by Mr. Anderson and Ms. 
Lopez during July and August 2003 were combined to produce a profile for PD activity 
A. With this activity level profile, we can describe each activity’s duration (e.g., total 
contact hours, mean contact hours, and span), type, purpose, and other quality features 
such as a mean level of active learning. 

Exhibit 21: Activity-level data aggregated across months and teachers 

(hypothetical data) 



Activity 


Jul- 

03 


Aug- 

03 


Sep- 

03 


Oct- 

03 


Nov- 

03 


Dec- 

03 


Jan- 

04 


Feb- 

04 


# of logs 
aggregated 


A 


2 


2 


0 


0 


0 


0 


0 


0 


4 


B 


0 


0 


1 


0 


1 


0 


1 


0 


3 


C 


0 


0 


0 


1 


0 


0 


0 


0 


1 


D 


0 


0 


0 


0 


1 


1 


1 


1 


4 


E 


1 


2 


2 


0 


0 


0 


0 


0 


5 


F 


0 


0 


1 


1 


0 


0 


0 


0 


2 


G 


0 


0 


0 


0 


1 


0 


0 


0 


1 



Third, we can produce teacher level data by aggregating the basic monthly log level data 
across months and activities. As the last column of Exhibit 22 demonstrates, a total of 20 
monthly logs were aggregated to produce 5 different teacher PD profiles (or 5 sets of 
teacher level data). For example, 6 monthly logs created by Mr. Anderson during the 8- 
month period were combined to produce his PD profile. With this teacher level profile, 
we can describe each teacher’s overall and cumulative PD experience such as duration 
(e.g., total contact hours, mean contact hours, and span), cumulative content focus, and 
other quality features such as a mean level of active learning and coherence. For example, 
Ms. Lopez’s profile seems to indicate that her engagement in PD is more distributed over 
time than that of Mr. Lee. 

Exhibit 22: Teacher-level data aggregated across months and activities 

(hypothetical data) 





Jul- 


Aug- 


Sep- 


Oct- 


Nov- 


Dec- 


Jan- 


Feb- 


# of logs 


Teacher 


03 


03 


03 


03 


03 


03 


04 


04 


aggregated 


Mr. Anderson 


1 


1 


1 


1 


1 


0 


1 


0 


6 


Ms. Lopez 


1 


1 


0 


0 


1 


1 


1 


1 


6 


Mrs. Kelly 


0 


1 


1 


0 


0 


0 


0 


0 


2 


Mr. Lee 


1 


1 


2 


1 


0 


0 


0 


0 


5 


Mrs. Smith 


0 


0 


0 


0 


1 


0 


0 


0 


1 
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In sum, the PDAL data represent a complex dataset that is multilevel or hierarchical in 
nature. Data from monthly logs are nested within activities and individual teachers.^ 
Furthermore, teachers are nested within schools, which, in turn, are nested within school 
districts, which, in turn, are nested within MSP program sites. In the remainder of the 
section, we will use teacher level data solely to form a MSP site-specific profile to allow 
for cross-site comparisons on key quality features. Results on the basic monthly log level 
data analyses will be shown in Appendix D. 

Construction of PD Scales 

Since the PDAL was specifically designed to depict professional development activities 
according to research-based criteria of quality (Caret et al., 1999), we created a number 
of variables and scales to capture such quality features. As was briefly described in 
Section I, there are three structural features and three core features that are associated 
with high quality professional development. They are: 

Three structural features 

• Duration (e.g., contact hours and span) 

• Type 

• Collective participation 
Three core features 

• Active learning 

• Coherence 

• Content focus 

Duration of PD Activity 

As stated in the No Child Left Behind Act, high quality professional development 
activities should provide, among others, “sustained and intense” learning experiences. To 
estimate how sustained the professional development activities are, we asked teachers 
about two aspects of duration to assess the extent to which teachers were provided with 
sustained activities: the number of contact hours spent in the PD activity for the month 
and the duration within the given month over which the activity was conducted (ranging 
from 0=less than a day, l=one day, 2=2-4 days, 3=a week, to 4=entire month). In 
addition, for continuous activities which extend beyond a given month, we created a new 
variable called span, which represents the period over which the sessions or components 
of the activity were spread."^ We operationally defined the span by the number of 
inclusive months from the starting to the ending month. 



^ To use a technical term, monthly logs are cross-classified (or cross-nested) within activities and individual 
teachers (Raudenhush and Bryk, 2002). 

We plan to create a new duration variable by combining contact hours, duration within a given month, 
and span. This new variable should be able to indicate true duration over the entire study period with given 
usable non-missing data. This duration variable may range from less than a day, a day, 2-4 days, a week, 2 
weeks to a month, 2-4 months, 5-7 months, 8 months to a year, to a year or longer. 
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Exhibit 23 displays our results for a number of duration-related variables such as contact 
hours and span. The results indicate that teachers vary widely in the amount of time they 
invested in their professional development activities. For example, an average amount of 
contact time per teacher per activity per month (Act_Hours) reported among 273 teachers 
was 16.89 hours with a standard deviation of 15.78.^ To compute the average contact 
hours, we summed the contact hours reported for each of the monthly logs that a teacher 
filled out over the period of the study (producing the total contact hours for the teacher); 
then divided the total contact hours by total number of completed monthly logs (i.e., all 
the non-zero hour activity logs) (producing a teacher’s mean contact hours per monthly 
log); and then finally averaged mean contact hours across teachers. For example, if Mr. 
Anderson reported 20, 20, 12, 12, 12, and 8 hours, respectively, for his 6 monthly logs, 
then his total contact hours would be 84 hours. And his mean contact hours is 14 (84 
divided by 6). It is important to note that we are reporting the mean number of contact 
hours for months in which teachers participated in PD; it’s not the mean for all months, 
which would be substantially lower, since we know that the teachers didn’t have any PD 
in some of those months. 

Since we expected some monthly or seasonal variation in time spent on PD activities, we 
computed average contact hours for each of the months during the study period. As 
Exhibit 23 shows, in fact, teachers were much more active in PD during the summer than 
during the winter. For example, the average contact hours for July 2003 and 2004 were 
37.04 and 31.72, respectively, while those for December 2003 and January 2003 were 
10.66 and 9.69, respectively. 

It is also important to note that we report the average amount of contact hours 
(Act_Hours) instead of the total amount of contact hours in this report. If we reported the 
total amount of contact hours, which is a sum of the hours for all months reported, it 
would be subject to error since some teachers completed many months of log and some 
completed few.^ Therefore, we computed the average amount of contact hours by 
dividing the sum of contact hours for all reported months by the number of months being 
reported. 



^ The results from the log-level data analysis (shown in Appendix D) are different from those from the 
teacher-level data analysis. For instance, the grand mean of entire activity contact hours drawn from entire 
1,798 monthly logs reported hy 273 teachers (15.37 hours) is not equal to the mean of mean contact hours 
for the 273 teachers (16.89 hours). This is because the number of logs completed by teachers is not the 
same. Some teachers filled out more logs than others. Given this unbalanced teacher representation, the 
grand mean of contact hours will be biased toward the mean contact hours reported by those teachers who 
completed more logs. It is common that the mean of means has smaller standard deviation than that of the 
grand mean. Generally, there is not much difference between these two mean figures. 

® At the time of this writing, there are a great number of logs which are yet to be completed and “closed” by 
teachers. These “open” or yet-to-be-closed logs may turn out to be “inactive” monthly reports or missing 
data; alternatively they may be filled in with activities at the last minute close to the end of the PDAL data 
collection. Until the PDAL data collection is entirely over, we cannot determine how to deal with these 
open status logs. On this ground, we included in our analysis monthly log data that show “active” activities. 
In due time, when we take into account “inactive” months (i.e., zero hour for the month), the estimate of 
the average contact hours will be adjusted. 
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Since the “duration within a given month” variable was not quite a precise interval 
measure, we created a dummy variable (ActD_Duration) which indicates whether an 
activity extended at least two days for the month. The result indicates that the percentage 
of activity logs that extended at least two day was 58%. Another variable that relates to a 
criterion of sustained activities is whether or not teachers were engaged in a continuous 
activity that extends beyond a given month (Act_Continued) rather than in a one-time 
event activity that ends within a single month. The result shows that overall 63% of the 
activities were continuous. For continuous activities, we computed the span of the 
activities (Act_Span) as well as the sheer number of months for which logs were created 
(Act_Mo_N). The result indicates that continuous activities were spread on average over 
a span of 2.76 months, while they involved on average 2.16 months. Mean and standard 
deviation for duration variables (e.g., contact hours, duration within a given month, and 
span) are listed in Exhibit 23 for teacher-level data. 
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Exhibit 23: Descriptive Statistics for PDAL Variables: Based on Teacher-level Data 



Variable 


Description 


N 


Mean 


SD 


Min 


Max 


Activity Duration 


ActD Duration 


Activity lasted for 2-4 days or longer: 0=one day or less, 1 = 2-4 days or longer 


273 


0.58 


0.36 


0 


1 


Act Hours 


Mean contact hours 


273 


16.89 


15.78 


1.0 


105.0 


Act Continued 


Activity is continued (0=no, 1 =yes) 


273 


0.63 


0.37 


0.0 


1.0 


Act Mo N 


Mean number of months 


273 


2.16 


1.79 


1.0 


11.2 


Act Span 


Mean activity span 


273 


2.76 


2.44 


1.0 


14.0 


Act Consecutive 


Activity occurs in consecutive months 


223 


0.67 


0.40 


0.0 


1.0 


mean Act Hour307 


Mean Activity Contact Hour during July 2003 


150 


37.04 


29.60 


1.0 


150.0 


mean Act HourSOS 


Mean Activity Contact Hour during August 2003 


150 


14.85 


16.23 


1.0 


80.0 


mean Act Hour309 


Mean Activity Contact Hour during September 2003 


117 


13.71 


19.99 


1.0 


160.0 


meanAct Hour310 


Mean Activity Contact Hour during October 2003 


148 


12.62 


17.10 


1.0 


160.0 


meanAct Hour311 


Mean Activity Contact Hour during November 2003 


117 


12.39 


18.52 


1.0 


160.0 


meanAct Hour312 


Mean Activity Contact Hour during December 2003 


96 


10.66 


16.23 


1.0 


140.0 


meanAct Hour401 


Mean Activity Contact Hour during January 2004 


98 


9.69 


17.26 


1.0 


160.0 


meanAct Hour402 


Mean Activity Contact Hour during February 2004 


101 


11.46 


12.92 


1.0 


80.0 


meanAct Hour403 


Mean Activity Contact Hour during March 2004 


96 


11.12 


11.09 


1.0 


60.0 


meanAct Hour404 


Mean Activity Contact Hour during April 2004 


62 


13.25 


13.06 


1.5 


70.0 


meanAct Hour405 


Mean Activity Contact Hour during May 2004 


74 


12.38 


10.19 


2.0 


60.0 


meanAct Hour406 


Mean Activity Contact Hour during June 2004 


49 


27.00 


20.35 


1.0 


80.0 


meanAct Hour407 


Mean Activity Contact Hour during July 2004 


49 


31.72 


19.58 


3.0 


80.0 


meanAct Hour408 


Mean Activity Contact Hour during August 2004 


25 


23.01 


16.57 


2.5 


60.0 


PD Quaiity Measures 


ActiveLearn 


Active learning (mean score): 0=never, 1=rarely, 2=sometimes, 3=often 


270 


1.14 


0.60 


0.0 


3.0 


Coherence 


Coherence (mean score): 0=never, 1=rarely, 2=sometimes, 3=often 


268 


2.27 


0.50 


0.5 


3.0 


Collective 


Collective participation at a given month (sum score): On the scale of 0 - 2: number 
of postive response to two collective participation items 


269 


0.72 


0.58 


0.0 


2.0 


Collective Overall 


Overall collective participation (sum score) 


269 


1.28 


0.78 


0.0 


2.0 


Content Focus Measures 


Purpose of PD 


Purp Subject 


Strengthening subject matter knowledge 


262 


0.75 


0.31 


0.0 


1.0 


Math Topics 


Topics math 


Total number of 1st-order topics covered in the activity 


151 


2.27 


1.53 


0.0 


6.0 


Intensetopic math 


Math topic intensity (contact hours divided by number of 1st-order topics) 


151 


5.41 


8.55 


0.0 


80.0 


Math instructionai Activities 


ProblemRoutine 


Routine math exercise/problem-solving: estimate, predict, apply, analyze, infer 


139 


0.67 


0.26 


0.0 


1.0 


Problem_Extended 


Extended problem-solving 


139 


0.38 


0.30 


0.0 


1.0 


M_Perform_Procedure Perform procedures (math) 


142 


0.49 


0.24 


0.0 


1.0 


M_Make_Connection 


Make connections (math) 


139 


0.62 


0.24 


0.0 


1.0 


IA1 math 


Number of broad instructional activities focused 


151 


3.02 


1.29 


0.0 


5.0 


IA2_math 


Number of specific instructional strategies focused within broad instructional 
activities covered 


143 


13.64 


7.48 


0.0 


35.0 


Science Topics 


topics science 


Total number of 1st-order topics covered in the activity 


125 


5.00 


4.28 


0.0 


24.5 


intensetopic science 


Science topic intensity (contact hours divided by number of 1st-order topics) 


125 


5.06 


6.24 


0.0 


36.3 


Science instructionai Activities 


Lab Work 


Lab-based activities 


111 


0.71 


0.25 


0.0 


1.0 


Info Collect 


Collecting science data/info outside lab 


112 


0.70 


0.27 


0.0 


1.0 


S Perform Procedure Perform procedures (science) 


114 


0.66 


0.24 


0.0 


1.0 


S Make Connection 


Make connections (science) 


111 


0.62 


0.30 


0.0 


1.0 


I A1 science 


Number of broad instructional activities focused 


122 


3.00 


1.28 


0.0 


5.0 


IA2_science 


Number of specific instructional strategies focused within broad instructional 
activities covered 


119 


14.80 


8.20 


0.0 


34.0 
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Lastly, it is important to note that currently the PDAL has only limited capacity to deal 
with certain types of censored observations. Censoring exists when incomplete 
information is available about the duration of professional development activities because 
of limited observation period. In other words, if PD activities started and ended within the 
PDAL period, the duration of the PD activities is unequivocally defined (without 
censored observation). However, if PD activities started either before the beginning of the 
PDAL data collection or PD activities continued after the end of the PDAL data 
collection, we would have censored observations. Exhibit 20 depicts a number of distinct 
situations regarding censored observations. In case of activities A and E, we are missing 
some unknown months of span for activities that began prior to the first log the teacher 
completed July 2003. On the other hand, in case of activity D (or possibly B), we are also 
missing some unknown months of span for activities that were still underway at the time 
of the last log the teacher submitted. Anticipating these potential problems, the PDAL 
asked teachers if it is their last month of participation in that activity. If it was their last 
month, the PDAL will provide uncensored observation. However, if it wasn’t their last 
month, then we will have right-censored activity. The true duration of that activity would 
remain unknown. On the other hand, we failed to ask teachers if they were engaged in 
that activity prior to the first month of the PDAL. If they were, then we would have left- 
censored observations. We should have asked teachers for how long they have been 
engaged in the activity before they first reported on the activity. We plan to refine the 
PDAL to rectify this limitation. 

Activity Type 

Some types of PD activities are more common than others. And particular types of 
activities may be more conducive to effective learning and may have lasting effects on 
teachers’ instructional practice. The PDAL asked the following question to identify the 
type of PD activity that teachers attended: 

Which of the following best describes the activity? If more than one response fits the 
activity, pick the response that best describes the aspect of the activities in which you 
spent the most time. (Choose only one response): 

□ Participation in a workshop or in-service activity 

□ Participation in a summer institute 

□ Attendance at a college course 

□ Attendance at a conference 

□ Participation in a teacher study group 

□ Participation in a teacher network or collaborative of teachers 

□ Working with a mentor, coach, lead teacher, or observer 

□ Participation in a teacher committee or task force 

□ Engagement in informal self-directed learning 

Despite a lengthy instruction on how to choose a single most appropriate response, many 
teachers picked more than one activity type, making the comparison of the prevalence of 
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each type difficult. Despite this complication, it is apparent that participation in workshop 
or in-service activity was the most prevalent type of activity teachers took part in. Of 
1,021 unique activities, 58% of them were workshop or in-service activities.^ 

Participation in a teacher network or collaborative of teachers is a distant runner up 
(27%), followed by working with a mentor, coach, lead teacher, or observer (17%), 
attendance at a college course (16%), participation in a teacher study group (16%), and 
engagement in informal self-directed learning (16%). Attendance at a conference was the 
least prevalent type of activity (8%). Note that we asked about the activity type only on 
the first month in which a teacher participated in a particular activity. We assumed the 
type would remain the same across months. 

Collective Participation 

Collective participation relates to opportunities for teachers from the same educational 
setting to engage in joint professional development. To assess the level of collective 
professional learning opportunity, the PDAL asked the following two questions with a 
two-point scale of 0=no and l=yes: 

Teachers may participate in professional development activities alone or with groups of 
teachers from their school. For this professional development activity, with whom did 
you participate? 

□ I participated with most or all of the teachers from my school. 

□ I participated with most or all of the teachers from my department or grade level. 

The scale of collective participation was created by combining these 2 items and 
computing a sum score. As Exhibit 24 shows, the reliability of the active learning scale, 
as indexed by Cronbach’s alpha, was .54. As Exhibit 23 indicates, the average level of 
collective participation (Collective) was .72, which indicates that many teachers tend to 
participate in professional development alone rather than in group. 

It is noteworthy that unlike active learning or coherence, collective participation, along 
with duration of activity, represents one of the structural features of PD. In other words, 
even if teachers answer differently from month to month about whether they participated 
with most or all of the teachers from their school or from department or grade level, the 
overall structure of the activity should stay the same over the span of the activity. For 
example, for a given month, teacher A reports that he participated in activity 1 with most 
of teachers from his school. In the next month, however, he may report otherwise about 
collective participation for the same activity. In this case, one may argue that even though 
teacher A’s perception or report has changed from one month to another, the very 
structure of activity I that incorporated the nature of collective participation must remain 
the same over the span of the activity. For this reason, we created a new scale 
(Collective_Overall) that reflects an overall collective participation over the span of 



^ Even though there are 1,798 monthly logs completed, some of them were filled out for the same activities 
over months. For the same activities, we did not repeat a question about activity type. Therefore, the PD 
type variable is based on 1,021 unique activities. 
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activity to distinguish it from collective participation at a given month (Collective). We 
operationally defined the overall eolleetive participation as having experienced collective 
participation at least one time over the span of the activity. The result shows that the 
average overall collective participation is 1.28, which is higher than the average of 
eolleetive partieipation measured at given months (.72). 

Active Learning 

Aetive learning coneerns the opportunities provided for teachers to beeome aetively 
engaged in meaningful discussion, planning, and practice as part of the professional 
development activity. To measure active learning, the PDAL asked the following 8 
questions with a four-point scale of 0=never, l=rarely, 2=sometimes, and 3=often: 

During this professional development activity this month, how often did you: 

□ Observe demonstrations of teaching techniques? 

□ Lead group diseussions? 

□ Develop curricula or lesson plans, which other participants or the activity leader 
reviewed? 

□ Review student work or score assessments? 

□ Develop assessments or tasks as part of a formal professional development 
activity? 

□ Practice what you learned and receive feedback as part of a professional 
development activity? 

□ Receive coaehing or mentoring in the classroom? 

□ Give a lecture or presentation to colleagues? 

The scale of active learning (ActiveLearn) was created by combining these 8 items and 
computing a mean seore. As Exhibit 24 shows, the reliability of the active learning seale, 
as indexed by Cronbach’s alpha, was .85. And as Exhibit 23 demonstrates, the average 
level of active learning was 1.14 (with a standard deviation of .60), which indicates that 
opportunities for active learning provided for teachers were quite low. 
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Exhibit 24: Factors and Reliability of Professional Development Quality Constructs: 

Based on Teacher-level Data 



Collective Participation In a Given Month 



Variable 


Label 


Factor 1 




i1490_1 


I participated with most or all of the teachers from my school. 






i1490_2 


I participated with most or all of the teachers from my department or grade level. 








alpha with all items 


0.54 




Overall Collective Participation 






Variable 


Label 


Factor 1 




collect_sch 


I participated with most or all of the teachers from my school. 


0.84 




collect_dept 


I participated with most or all of the teachers from my department or grade level. 


0.84 






alpha with all items 


0.60 




Active Learning 








Variable 


Label 


Factor 1 


Factor 2 


i1488_1 


Observe demonstrations of teaching techniques? 


-0.02 


0.86 


i1488_3 


Lead group discussions? 


0.51 


0.56 


i1488_5 


Develop curricula or lesson plans, which other participants or the activity leader reviewed? 


0.80 


0.22 


i1488_6 


Review student work or score assessments? 


0.83 


0.05 


i1488_7 


Develop assessments or tasks as part of a formal professional development activity? 


0.84 


0.23 


i1488_8 


Practice what you learned and receive feedback as part of a professional development activil 


0.50 


0.57 


i1488_9 


Receive coaching or mentoring in the classroom? 


0.21 


0.68 


i1488_10 


Give a lecture or presentation to colleagues? 


0.53 


0.50 




alpha with all items 


0.85 




Coherence 








Variable 


Label 


Factor 1 




i1489_1 


Designed to support the school-wide improvement plan adopted by your school? 


0.73 




i1489_2 


Consistent with your department or grade level plan to improve teaching? 


0.86 




i1489_3 


Consistent with your own goals for your professional development? 


0.84 




M489 4 


Based explicitly on what you had learned in earlier professional development activities? 


0.69 




i1489_5 


Followed up with related activities that built upon what you learned as part of the activity? 


0.82 






alpha with all items 


0.83 
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Exhibit 24: Factors and Reliability of Professional Development Quality Constructs: 
Based on Teacher-level Data (Continued) 



Routine Math Problem-Solving 



Variable 


Label 


Factor 1 




i1691_3 


Students Work Individually - Explain their reasoning or thinking in solving a problem, using 
several sentences orally or in writing 


0.58 




i1691_4 


Students Work Individually - Apply mathematical concepts to real-world problems 


0.72 




i1691_5 


Students Work Individually - Make estimates, predictions or hypotheses 


0.86 




M691 6 


Students Work Individually - Analyze data to make inferences or draw conclusions 


0.86 




i2160_3 


Students Work in Pairs or Small Groups - Talk about their reasoning or thinking in solving a 
problem 


0.79 




i2160_4 


Students Work in Pairs or Small Groups - Apply mathematical concepts to real-world 
problems 


0.80 




12160 5 


Students Work in Pairs or Small Groups - Make estimates, predictions or hypotheses 


0.89 




i2160_6 


Students Work in Pairs or Small Groups - Analyze data to make inferences or draw 
conclusions 


0.85 






alpha with all items 


0.91 




Extended Math Problem-Solving 






Variable 


Label 


Factor 1 


Factor 2 


i1691_7 


Students Work Individually - Work on a problem that takes at least 45 minutes to solve 


0.94 


0.16 


i1691_8 


Students Work Individually - Complete or conduct proofs or demonstrations of their 
mathematical reasoning 


0.23 


0.89 


i2160_7 


Students Work in Pairs or Small Groups - Work on a problem that takes at least 45 minutes 
to solve 


0.91 


0.24 


i2160_8 


Students Work in Pairs or Small Groups - Complete or conduct proofs or demonstrations of 
their mathematical reasoning 


0.16 


0.91 




alpha with all items 


0.80 




Laboratory-based Work 






Variable 


Label 


Factor 1 




i2085_1 


Lab Activities, Investigations, or Experiments - Make educated guesses, predictions, or 
hypotheses 


0.55 


0.55 


i2085_2 


Lab Activities, Investigations, or Experiments - Follow step-by-step directions 


0.64 


0.10 


i2085_3 


Lab Activities, Investigations, or Experiments - Use science equipment or measuring tools 


0.70 


0.12 


i2085_4 


Lab Activities, Investigations, or Experiments - Collect data 


0.70 


0.29 


i2085_5 


Lab Activities, Investigations, or Experiments - Change a variable in an experiment to test a 
hypothesis 


0.25 


0.77 


i2085_6 


Lab Activities, Investigations, or Experiments - Organize and display information in tables or 
graphs 


0.24 


0.69 


i2085_7 


Lab Activities, Investigations, or Experiments - Analyze and interpret science data 


0.66 


0.45 


i2085_8 


Lab Activities, Investigations, or Experiments - Design their own investigation or experiment 
to solve a scientific question 


0.12 


0.83 


i2085_9 


Lab Activities, Investigations, or Experiments - Make observations/classifications 


0.80 


0.29 




alpha with all items 


0.86 




Collecting Science Data/Information Outside the Lab 






Variable 


Label 


Factor 1 




i2087_1 


Collecting Science Data or Information - Have class discussions about the data 


0.87 




i2087_2 


Collecting Science Data or Information - Organize and display the information in tables or 
graphs 


0.69 




i2087 3 


Collecting Science Data or Information - Make a prediction based on the data 


0.67 




i2087_4 


Collecting Science Data or Information - Analyze and interpret the information or data, orally 
or in writing 


0.86 




i2087_7 


Collecting Science Data or Information - Make a presentation to the class on the data, 
analysis or interpretation 


0.59 






alpha with all items 


0.78 
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Coherence 

Coherence concerns the extent to which professional development activities are perceived 
by teachers to be a part of a coherent program of teacher learning. To gauge the level of 
coherence of PD activities that teachers participated in, the PDAL asked the following 5 
questions with a five-point scale of 0=never, l=rarely, 2=sometimes, 3=often, and 
9=N/A: 

How often was this professional development activity? 

□ Designed to support the school- wide improvement plan adopted by your school? 

□ Consistent with your department or grade level plan to improve teaching? 

□ Consistent with your own goals for your professional development? 

□ Based explicitly on what you had learned in earlier professional development 
activities? 

□ Followed up with related activities that built upon what you learned as part of the 
activity? 

The scale of coherence was created by combining these 5 items and computing a mean 
score. As Exhibit 24 displays, the reliability of the active learning scale, as indexed by 
Cronbach’s alpha, was .83. The average level of coherence was 2.27 (with a standard 
deviation of .50), which indicates that teachers experienced quite coherent professional 
development across activities and over time. 

Content Focus 

Content focus concerns how much emphasis was given in a PD activity on enhancing 
content knowledge and skills. We took a number of different approaches to collect 
information about content focus as well as content coverage. First, the PDAL asked 
teachers a single-item question asking directly about whether strengthening subject 
matter knowledge was a purpose of the PD activity they took part in. On this measure 
(Purp_Subject), PD was reported to be focused on content in 74% of all activity logs (See 
Exhibit 23). 

Second, to examine on which specific content areas and topics were being focused, the 
PDAL presented to teachers a list of topical areas (separately for math or science) and 
asked them which topical areas were a primary focus in their professional development 
activity. Eor example, topical areas in math included: 

□ Number sense/Properties/Relationships 

□ Measurement 

□ Algebraic Concepts 

□ Geometric Concepts 

□ Operations 

□ Data Analysis/Probability/Statistics 
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What were the most common content topical areas focused in PD activities for math and 
science? In math, algebraic concepts were most often the main focus of PD, wh i le 
measurement was least often focused: 40% vs. 27% of reported logs (see Table D1 in 
Appendix D). In science, nature of science was most frequently covered in PD, while 
nuclear chemistry was on the bottom of the list: 38% vs. 9% (See Table Dl). 

As a basic descriptive measure of content coverage (not content focus), we created a 
variable - separately for math and science - which represents the number of topical areas 
that were covered as a primary focus in a given activity.^ As Exhibit 23 shows, on 
average math teachers were offered 2.27 topical areas in their PD given the 6 maximum 
areas. More topical areas were presented in science, which include such a wide array of 
topical areas as nature of science, botany, animal biology, human biology, evolution, 
energy, motion and forces, astronomy, nuclear chemistry - totaling 25 different topical 
areas. On average, science teachers were exposed to 5 topical areas in their PD. 

For each topical area that was marked as being a primary focus in their PD activity, we 
then asked teachers which specific topics were of primary focus in their PD activity. For 
example, for the topical area of number sense/properties/relationships, we listed a number 
of specific topics that range from place value and whole numbers, to patterns and real 
number, and to mathematical properties. For example, within the topical area of number 
sense/properties/relationships, on average 3.28 topics were focused in math teachers’ PD 
(See Exhibit 23). 

As another related measure of content focus, we created a variable called topic intensity, 
which is operationally defined as contact hours divided by the number of topical areas 
being primarily focused in a given activity. In math, 5.41 hours were devoted per a 
focused topic, while 5.06 hours in science (See intensetopic_math and 
intensetopic_science in Exhibit 23). 

Third, as a measure of content focus, we examined how many topics on instructional 
activities and strategies were covered and highlighted in PD. For example, in science, the 
PDAL asked teachers whether their PD was focused on any of the following broad 
instructional activities for use in their classroom: 

□ Laboratory Activities, Investigations, or Experiments 

□ Classroom Activities When Students Work in Pairs or Small Groups 

□ Collecting Science Data or Information 

□ Use of Calculators, Computers, or Other Educational Technology 

□ Assessments 

And then for each instructional activity that was marked as being a primary focus in their 
PD, we further asked teachers which specific instructional strategies were a primary 



^ While content focus is part of core features associated with high quality professional development, 
content coverage is not. In other words, it is not necessarily desirable to have a more content coverage in 
PD. We created content coverage variables as an intermediate step to get at another measure of content 
focus, namely, topical intensity (which is discussed shortly). 
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focus in their PD activity. For example, for the instructional activity area of laboratory 
activities, investigations, or experiments, the PDAL asked the following questions: 

Did the PD focus on any of the following instructional strategies for use in your 
classroom (relating to laboratory activities, investigations, or experiments)? 

□ Make educated guesses, predictions, or hypotheses 

□ Follow step-by-step directions 

□ Use science equipment or measuring tools 

□ Collect data 

□ Change a variable in an experiment to test a hypothesis 

□ Organize and display information in tables or graphs 

□ Analyze and interpret science data 

□ Design their own investigation or experiment to solve a scientific question 

As a basic descriptive measure of content coverage (not content focus), we created a 
variable - separately for math (IAl_math) and science (IAl_science) - that represents the 
number of instructional activities that were covered as a primary focus in a given PD 
activity. In addition, we counted the number of specific instructional strategies within 
broad instructional activities that were covered as a primary focus in a given PD - 
separately for math (IA2_math) and science (IA2_science). As Exhibit 23 indicates, on 
average 3 out of 5 broad instructional activities were of a primary focus in PD in math 
and science alike. Given instructional activities, 13.6 specific instructional strategies 
came to being focused in math, while 14.8 in science. 

The quantity of instructional activities or strategies that are focused on in PD may not be 
useful in shedding light on whether teachers experienced high quality content focus. To 
differentiate the qualitative differences among instructional strategies being covered and 
focused in PD, we conducted an exploratory factor analysis. Among a number of factors 
that merged from the analysis of 35 instructional strategies covered in math, we identified 
two factors which we considered represent qualitatively distinct instructional strategies. 
We created two scales based on the factors: routine math problem-solving and extended 
math problem-solving. As Exhibit 24 demonstrates, the internal consistency among the 
variables within each of the two scales was high: .91 and .80 for routine math problem- 
solving and extended math problem-solving strategies, respectively. The routine math 
problem-solving scale reflects routine strategies commonly used in math instruction such 
as reasoning, applying concepts to real-world problems, making estimate, and analyzing 
data to make inferences; while the extended math problem solving scale represents 
somewhat more extensive (and possibly more in-depth) problem-solving such as working 
on a problem that takes at least 45 minutes to solve and completing or conducting proofs 
or demonstrations of mathematical reasoning. As can be seen in Exhibit 23, teachers were 
less exposed to extended problem-solving instructional strategies in their PD than to 
routine problem-solving instructional strategies: on average, 38% vs. 67% of their 
activities, respectively. 

Using a similar procedure, we created two instructional strategy scales for science: 
laboratory-based work and science data/information collection outside the lab. The 
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laboratory-based work scale involves an array of activities that are required to conduct 
lab or experiment based investigation such as generating hypotheses, following 
instructions, using equipments, collecting data, and analyzing data; the science 
data/information collection scale encompasses a host of data collection activities that take 
place outside the lab such as having class discussion about science data, organizing and 
displaying science data/information in tables and graphs, and making presentation to the 
class on the data, analysis or interpretation. As Exhibit 24 demonstrates, the internal 
consistency among the variables within each of the two scales was high: .86 and .78 for 
the scales of laboratory-based work and science data/information collection outside the 
lab, respectively. As Exhibit 23 shows, teachers reported that their PD was focused in 
these two strategies about equally: on average, about 70% in each of the two instructional 
strategies. 

Lastly, we created two more scales that are based on specific instructional strategies - 
separately for math and science - to replicate the SEC’s cognitive demand based scales: 
performing procedures and making connections (see Section III for items making up 
these two scales). 

Analysis of Professional Development during the Project Period 

Data from the Professional Development Activity Log are used to test Hypotheses 3 and 
4 for the study. 

• Hypothesis 3: In year 2, the initial year of MSP implementation, we expect sites 
to differ in the quality of the professional development teachers experienced. 

• Hypothesis 4: In year 2, we expect teachers that in the treatment group (i.e., 
teachers targeted for MSP participation) experienced higher quality professional 
development than teachers in the comparison group. 

Specifically, we examine whether teachers’ PD experiences were similar or different in 
each of the following six quality features of professional development activities that were 
described in Sections II and IV. Analysis was conducted separately for data on math and 
science teachers as they employ different instructional strategies and experience different 
professional development. 

Differences in PD among MSP Program Sites 

Mathematics Professional Development. We hypothesized that since each MSP 
program site emphasizes and implements different strategies to accomplish their program 
goals, some program-related variation between the four MSP sites may exist in terms of 
the quantity and quality of professional development that is being offered. 

Consistent with our hypothesis, we found some differences in the quality of math PD 
between MSP program sites. First, as the top panel of Exhibit 25 indicates, teachers 
sampled from different MSP sites experienced PD activities of varying duration, mean 
contact time, and mean activity span. For example, teachers who are affiliated with the 
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sites 1 and 2 - treatment and comparison teachers alike - attended PD with substantially 
longer in duration and with more mean contact hours than their counterparts in the two 
other sites. Site 1 teachers spent on average about twice as much time in PD as site 4 
teachers did (22.5 vs. 11.9 hours). Further, site 1 teachers’ activity span was a little more 
than twice as long as that of site 4 teachers’ (3.3 vs. 1.5 months). 

Second, we found no significant difference in opportunities for active learning among the 
4 MSP sites. Third, we found some significant variation among the MSP sites in the 
degree to which teachers experienced coherent PD activities. Sites 1 and 3 teacher’s 
benefited slight advantage over other sites, particularly site 4 teachers. Fourth, whether 
we examined the overall collective participation variable or the variable of collective 
participation at a given month, there was no significant difference among the four MSP 
sites. 
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Exhibit 25: Quality of Math Professional Development Activities Compared by MSP Site and Treatment Status: 

Based on Teacher-level PDAL Data 



Category 


n' 


% Duration 
>2-4 days 


Mean 

contact 

hours 


Mean 

activity 

span 


Active 

learning Coherence 


Collective 

partici- 

pation 

(Overall) 


Collective 
participa- 
tion (At a 
given 
month) 


Strengthen 

subject 

matter 

knowledge 


Topic 

intensity 


Routine 

math 

problem 

solving 


Extended 

math 

problem 

solving 


Perform 

procedures 


Make 

connec- 

tions 


MSP Site 






** 




ns 


* 


ns 


ns 


ns 


ns 


* 


ns 


ns 


ns 


Site 1 


37 


0.67 


22.5 


3.3 




2.4 










0.58 








Site 2 


49 


0.73 


13.9 


3.7 




2.2 










0.64 








Site 3 


34 


0.40 


14.0 


2.1 




2.3 










0.76 








Site 4 


28 


0.46 


11.9 


1.5 




2.0 










0.74 








Treatment 




** 






ns 


ns 


ns 


* 


ns 


* 


ns 


ns 


ns 


ns 


Comparison 


45 


0.46 


11.2 


2.1 








0.86 




2.5 










Treatment 


103 


0.64 


17.7 


3.1 








0.62 




6.7 










MSP_Site*Treatment 




ns 




ns 


ns 


ns 


ns 


ns 


ns 


ns 


ns 


ns 


ns 


ns 



Site 1 -Comparison 


12 


10.7 


Site 1 -Treatment 


25 


28.2 


Site 2-Comparison 


14 


11.3 


Site 2-Treatment 


35 


15.0 


Site 3 -Comparison 


6 


18.2 


Site 3 -Treatment 


28 


13.1 


Site 4-Comparison 


13 


8.4 


Site 4-Treatment 


15 


14.9 



Note: ^ N may vary due to variable-wise missing data. 

ns denotes non-signifieanee, *<.05, **<.01, and ***<.001. 
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Lastly, we found a little evidence of significant variation in content focus between the 
MSP sites. Only in routine math problem-solving, MSP sites 3 and 4 were higher than the 
other two sites. However, the four sites equally emphasized extended math problem- 
solving or stressed subject matter knowledge in their PD. 

Science Professional Development. Consistent with our expectation, we also 
found some significant differences in the quality of science PD among MSP program 
sites. First, as the top panel of Exhibit 25 shows, teachers affiliated with different MSP 
sites experienced PD activities of varying duration and activity span. For example, as 
with math teachers, science teachers who came from sites 1 and 2 - treatment and 
comparison teachers alike - attended PD with significantly longer duration and span than 
their counterparts in sites 3 and 4. Site 2 science teachers’ activity span was over twice as 
long as that of site 4’s counterparts (3.6 vs. 1.5 months). However, in terms of mean 
contact time, all 4 MSP sites weren’t significantly different. 

Second, we found no significant difference in opportunities for active learning, 
coherence, and collective participation among the 4 MSP sites. 

Lastly, we found significant variation between the MSP sites in five of the eight content- 
related measures we analyzed. For example, site 1 teachers experienced the most intense 
science PD of all. Their average contact time per topical area (e.g., earth systems) was 9.4 
hours, whereas that of site 3 and 4 teachers was about 3.5 hours. In contrast, however, 
site 1 teachers were least likely to get any of four instructional activities and strategies 
emphasized in their PD: lab-based work, science data/information collection, performing 
procedures, and making connections. In all of these four measures of content focus, site 
2 scored highest of all four MSP sites. 

In summary, we found that the four MSP program sites differed from each other in a 
number of PD quality measures both in math and science. This finding is consistent with 
our expectation that each site implements different strategies for their program goals and 
institutes professional development activities of different qualities. However, it is 
important to take caution to interpret these results given the preliminary nature of these 
analyses that are based on incomplete data and scales and variables still at their 
exploratory stage. 
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Exhibit 26: Quality of Science Professional Development Activities Compared by MSP Site and Treatment Status: 

Based on Teacher-level PDAL Data 



Category 


N‘ 


% Duration 
>2-4 days 


Mean 

contact 

hours 


Mean 

activity 

span 


Active 

learning 


Coherence 


Collective 

participa- 

tion 

(Overall) 


Collective 
participa- 
tion (At a 
given 
month) 


Strengthen 

subject 

matter 

knowledge 


Topic 

intensity 


Lab-based 

work 


Science 

data/ 

informa- 

tion 

collection 


Peform 

procedures 


Make 

connec- 

tions 


MSP Site 






ns 




ns 


ns 


ns 


ns 


ns 




* 


** 


* 


* 


Site 1 


18 


0.69 




2.2 












9.4 


0.55 


0.53 


0.53 


0.48 


Site 2 


63 


0.66 




3.6 












4.9 


0.77 


0.76 


0.71 


0.69 


Site 3 


23 


0.39 




1.6 












3.6 


0.71 


0.70 


0.64 


0.57 


Site 4 


21 


0.40 




1.7 












3.5 


0.67 


0.64 


0.60 


0.54 


Treatment 






ns 




ns 


ns 


ns 


ns 








*** 






Comparison 


54 


0.48 




2.2 










0.61 


3.6 


0.64 


0.61 


0.60 


0.52 


Treatment 


71 


0.64 




3.1 










0.81 


6.2 


0.76 


0.77 


0.69 


0.68 


MSP_Site*Treatment 




ns 


ns 


ns 


ns 


ns 


ns 


ns 


ns 


* 


ns 


ns 


ns 


ns 



Site 1 -Comparison 8 3.6 

Site 1 -Treatment 10 14.0 

Site 2-Comparison 15 4.0 

Site 2-Treatment 48 5.2 

Site 3 -Comparison 22 3.7 

Site 3 -Treatment 1 0.9 

Site 4-Comparison 9 2.9 

Site 4-Treatment 12 3.9 



Note: ^ N may vary due to variable-wise missing data. 

ns denotes non- significance, *<.05, **<.01, and ***<.001. 
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Differences in Professional Development between Treatment vs. 
Comparison Groups 

Mathematics Professional Development. At the outset, we hypothesized that if 
MSP initiatives are effectively implemented in each of the program sites, we may 
anticipate that compared with comparison teachers, treatment teachers would benefit 
more from professional development that is higher in quality (e.g., more opportunities for 
active learning, more coherent and intense activity). Moreover, as the effectiveness of 
program implementation is expected to vary from site to site, the relative advantage of 
treatment teachers over their comparison counterparts may also differ from site to site.^ 

As we expected, we found that the treatment teachers experienced math PD activities that 
are quantitatively and qualitatively somewhat different from that experienced by their 
comparison counterparts on a few accounts. First, as the middle panel of Exhibit 25 
indicates, the treatment teachers took part in PD activities that are characterized by a 
longer duration (e.g., more contact hours and longer activity span). For instance, 
treatment teachers had the mean contact time of 17.7 hours, while their colleagues in the 
comparison group had 11.2 hours. As the bottom panel of Exhibit 25 shows, however, the 
relative advantage of treatment group over comparison group was not equal across the 
four MSP sites. For example, the gap between the treatment vs. comparison groups was 
larger in site 1 (28.2 vs. 10.7 hours) than in site 2 (15.0 vs. 11.3 hours). Further, the 
general pattern of treatment group’s edge over comparison group was reversed in site 3, 
where comparison teachers reported more contact hours than their treatment counterparts: 
18.2 and 13.1 hours for comparison and treatment teachers, respectively. 

Second, we detected no significant difference either in opportunities for active learning or 
in coherence between the treatment and comparison teachers’ PD activities. Third, we 
found some significant difference between the treatment and comparison teachers in the 
degree to which teachers participate in PD activities collectively. Contrary to our 
expectation, the comparison teachers collectively attended math PD activities at a given 
month more often than their treatment group teachers did: 86% vs. 62% of time. 

However, there was no significant difference in terms of overall collective participation. 

It seems that MSP programs may not have relied on types of PD that required teachers’ 
collective participation such as district-wide or school-wide workshops. If MSP 
programs’ PD activities were instead primarily offered through college courses (that are 
presumably based on individual enrollment), we would expect that treatment teachers 
documented less collective participation. 

Lastly, inconsistent with our expectation, there was very little difference in content focus 
between the treatment teachers’ math PD and that of the comparison teachers. A 



^ Treatment teachers, by definition, take part in PD activities provided by any of the four MSP program 
sites. In addition, they may also participate in other PD activities that are offered by non-MSP programs. 
Therefore, we cannot attribute any difference between treatment and comparison teachers solely to the 
effect of MSP programs. It is quite possible that this difference may partly result from the lack of 
participation in PD activities on the part of the comparison teachers. 
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significant difference was detected only in one of several measures of content focus we 
gathered. The treatment teachers’ topie intensity registered 6.7 hours, while the 
comparison teachers did only 2.5 hours. Reeall that the topic intensity is a funetion of the 
mean contact hours and the number of topical areas focused. Since the treatment teachers 
had more contaet hours and fewer - if not significantly fewer - topieal areas focused in 
their PD than their comparison counters, the treatment teachers’ topic intensity was 
substantially higher. 

Science Professional Development. As was the ease in math, we found that the 
treatment science teachers received PD activities that are quantitatively and substantively 
different from that experienced by their comparison counterparts on several accounts. 
First, as the middle panel of Exhibit 26 indicates, compared to comparison teachers, 
treatment teachers attended extended and sustained PD aetivities (i.e., longer in duration 
and activity span). For example, 64% of treatment teachers participated in PD activities 
that are at least 2 days long in duration at a given month, while 48% of their colleagues in 
the comparison group took part in such long PD activities. However, in terms of mean 
contaet hours, the two groups were not different. 

Second, contrary to our expectation, we failed to find any significant difference between 
the treatment and comparison teachers in such important features of PD quality as active 
learning, coherence, and collective partieipation. It is interesting to note that sueh a lack 
of quality difference was also observed between the 4 MSP sites. It seems that such PD 
quality features are partieularly more difficult to institute eompared to other features sueh 
as duration or content focus. 

Lastly, we found a great deal of differences between the treatment and comparison 
teachers in their PD aetivities’ content focus. In all measures of content focus, we found 
that the treatment teachers took part in the PD activities that were significantly more 
focused on content. In general, compared to their non-MSP eounterparts, the MSP 
programs were more likely to provide PD activities that emphasize strengthening subject 
matter knowledge as its purpose, are more intense, and focus more on speeific 
instructional strategies. For example, in the use of specific instructional strategies such as 
lab-based work and seience data eolleetion outside the lab, the treatment teachers were 
more likely to report that their PD was focused on such strategies than their colleagues in 
comparison group did. 

This finding of a great deal of difference in the content focus of science PD activities 
between MSP programs and their non-MSP counterparts strikes an interesting contrast 
with only a slight difference in math PD. Recall that in math we found a signifieant 
difference only in one of six content focus measures (i.e., topic intensity) between 
treatment vs. comparison teachers, while in science we found sueh a signifieant 
difference in all measures. 

However, we need take caution in interpreting the results regarding differences between 
treatment vs. control groups given the preliminary nature of these analyses that are based 
on incomplete data and scales and variables still at their exploratory stage. 
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V. Illustration of Program Site-specific Effects 

In this section, we report the result of analysis that we conducted to test some potential 
effects which may be specific to a particular MSP program site. By testing program- 
specific hypotheses, we may be able to determine if individual MSP program is 
accomplishing its goal. A case in point for this report is the MSP Program site 1 that 
emphasized the use of educational technology in their professional development for their 
teachers. We intend to extend this type of program-specific analysis for the final report. 

For the site 1, we asked the following evaluation questions: 

1 . Is there any difference between the treatment and comparison teachers in the 
amount of instructional time they report spending with students in the use of 
computers, calculators, or other technology? 

2. Is there any difference in the professional development of the treatment and 
comparison teachers regarding the topical emphasis on the instructional use of 
computers, calculators, or other technology? 

With regard to the first question, we hypothesized that that there would be no reason why 
the treatment and comparison teachers should differ in their instructional use of 
technology in the baseline year unless the selection of the treatment teachers into the 
MSP program site 1 was based on that criterion (Hypothesis l).**^ With regard to the 
second question, given the MSP site I’s emphasis on the integration of technology to 
instruction, we hypothesized that, compared to their comparison counterparts, the 
treatment teachers would experience more professional development activities that stress 
the use of computers, calculators, or other technology (Hypothesis 2).** 

We analyzed Year 1 SEC data to address Hypothesis 1 regarding teachers’ instructional 
practice at the baseline. In addition, we analyzed the PDAL data to test Hypothesis 2 
about professional development activities that teachers attend during the project period. 

As Exhibit 3 shows, the total of 91 teachers participated in the MSP Program site 1 study, 
of which 48 teachers were “treated” by engaging in any of its MSP program activities 
(treatment teachers), while 43 were not (comparison teachers). Of 91 teachers, 78 
teachers completed Year 1 SEC while only 54 filled out PDAL. 

Consistent with Hypothesis 1, there was little significant difference between the treatment 
and comparison teachers in terms of their instructional time in the use of technology such 
as computer and calculators in the baseline year (See Exhibit 27). 

Consistent with Hypothesis 2, there was some significant difference between the 
treatment and comparison teachers in their professional development experiences, most 



This hypothesis is equivalent to Hypothesis 3 that was tested in Section IV. Since this test is conducted 
within a single MSP program, there is no need for hypotheses about difference between MSP sites, which 
were Hypotheses 1 and 2 in Section IV. 

This hypothesis is equivalent to Hypothesis 4 that was tested in Section IV. 



54 




MSP PD Study: Year 2 Progress Report 



likely due to the MSP program effect. Specifically, compared with their comparison 
counterparts, the treatment-group teachers experienced significantly higher amount of 
professional development activities thdX focused on the instructional use of computers, 
calculators, and other technology (see the second panel of Exhibit 28), even though the 
coverage of the topic involving such instructional technology did not differ significantly 
in each group’s professional development (see the first panel of Exhibit 28). 

Exhibit 27: Amount of Teachers' Instructional Activity Using Technology (Based on 
Year 1 SEC Data): By Subject Taught and By MSP Program Treatment Status 





Comparison 




Treatment 








Mathematics 


Mean^ 


SD 


N 


Mean^ 


SD 


N 


F-value 


2 

P 


How much of the total math instructional time do students in the target 
class use computers, calculators, or other technology to learn math?” 


3.45 


1.62 


19 


3.05 


1.34 


28 


0.83 


ns 


When students in the target class are engaged in activities that involve the use of calculators, computers, or other educational technology as part 


of mathematics instruction, how much time do they: 


















Learn facts 


2.08 


1.55 


19 


2.02 


1.74 


28 


0.02 


ns 


Practice procedures 


2.63 


1.50 


19 


2.89 


1.68 


28 


0.30 


ns 


Use sensors and probes 


0.53 


1.29 


19 


0.66 


1.19 


28 


0.14 


ns 


Retrieve or exchange data or information (for example, using the 
Internet or partnering with another class) 


0.74 


1.39 


19 


1.70 


1.70 


28 


4.16 


* 


Display and analyze data 


1.92 


1.85 


19 


2.27 


1.69 


28 


0.44 


ns 


Develop geometric concepts (for example, using simulations) 


0.95 


1.50 


19 


1.39 


1.54 


28 


0.97 


ns 




Comparison 




Treatment 








Science 


Mean^ 


SD 


N 


Mean ^ 


SD 


N 


F-value 


2 

P 


How much of the total science instructional time do students in the 
target class use computers, calculators, or other technology to learn 
science? 


2.35 


1.73 


17 


2.29 


1.44 


14 


0.91 


ns 


When students in the target class are engaged in activities that involve the use of calculators, computers, or other educational technology as part 


of science instruction, how much time do they: 


















Learn facts 


3.12 


1.36 


17 


3.07 


0.92 


14 


0.01 


ns 


Practice procedures 


3.12 


1.32 


17 


2.57 


1.34 


14 


1.30 


ns 


Use sensors and probes (for example, CBL's) 


1.06 


1.25 


17 


0.21 


0.58 


14 


5.42 


* 


Retrieve or exchange data or information (for example, using the 
Internet or partnering with another class) 


1.53 


1.46 


17 


1.79 


1.58 


14 


0.22 


ns 


Display and analyze data 


2.00 


1.34 


17 


2.08 


1.59 


14 


0.02 


ns 


Develop problems using simulations 


1.35 


1.46 


17 


1.57 


1.40 


14 


0.18 


ns 



Note: * The amount of instructional activity was measured on the scale of 0=none, l=little, 2=some, 3=moderate, 4=considerable, and 5=almost all. 
^ ns stands for non-signficanee, * <.05, ** <.01, and *** <.001. 



More specifically, compared to their comparison teachers, math treatment teachers were 
more likely to experience professional development activities that are focused on 
presumably high-order instructional strategies for use in their classroom with their 
students such as displaying and analyzing data or developing geometric concepts. But no 
such group difference was found in the professional development activities that focus on 
low-order instructional strategies such as learning facts or practicing procedures. Even 
though the general pattern of difference between the treatment and comparison teachers 
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was found similarly in math and science, the group difference was more noticeable in 
math than in science. 

It is important to note that we need to take these findings with some caution. For 
example, due to its small sample size, we may not have been able to detect any real 
significant difference between treatment and comparison groups. 

In sum, this test of hypotheses supports some evidence that the MSP program site I is 
accomplishing its goal of providing its teachers with high quality professional 
development activities that stress on the use of technology in math and science 
instruction. 

Exhibit 28: Teachers' Professional Development Experience in the Use of Technology 
(Based on PDAL Data): By Subject Taught and By MSP Program Treatment Status 





Comparison 




Treatment 








Mathematics 


Mean^ 


SD 


N 


Mean^ 


SD 


N 


F-value 


2 

P 


Was an instructional topic in the “use of computers, calculators, or 
other technology to learn math” covered in this professional 
development activity? 


0.64 


0.34 


12 


0.78 


0.22 


25 


2.29 


ns 


Did the professional development focus on any of the following instructional strategies for use in 


your classroom with your students (relating to 


calculators, computers, or other educational technology)? 


















Learn facts 


0.46 


0.40 


11 


0.38 


0.33 


25 


0.38 


ns 


Practice procedures 


0.60 


0.32 


11 


0.55 


0.35 


25 


0.13 


ns 


Use sensors and probes 


0.05 


0.15 


11 


0.32 


0.26 


25 


10.68 


** 


Retrieve or exchange data or information (for example, using the 
Internet or partnering with another class) 


0.17 


0.25 


11 


0.43 


0.29 


25 


6.74 


* 


Display and analyze data 


0.53 


0.37 


11 


0.76 


0.27 


25 


4.26 


* 


Develop geometric concepts (for example, using simulations) 


0.11 


0.20 


11 


0.47 


0.34 


25 


10.96 


** 




Comparison 




Treatment 








Science 


Mean^ 


SD 


N 


Mean^ 


SD 


N 


F-value 


2 

P 


Was an instructional topic in the “use of computers, calculators, or 
other technology to learn science” covered in this professional 
development activity? 


0.46 


0.41 


8 


0.71 


0.32 


9 


1.92 


ns 


Did the professional development focus on any of the following instructional strategies for use in 


your classroom with your students (relating to 


calculators, computers, or other educational technology)? 


















Learn facts 


0.60 


031 


6 


0.56 


0.41 


9 


0.03 


ns 


Practice procedures 


0.63 


0.31 


6 


0.55 


0.40 


9 


0.18 


ns 


Use sensors and probes (for example, CBL's) 


0.07 


0.10 


6 


0.54 


0.38 


9 


8.68 


* 


Retrieve or exchange data or information (for example, using the 
Internet or partnering with another class) 


0.72 


0.34 


6 


0.69 


0.31 


9 


0.03 


ns 


Display and analyze data 


0.70 


0.27 


6 


0.69 


0.34 


9 


0.00 


ns 


Develop problems using simulations 


0.10 


0.17 


6 


0.77 


0.24 


9 


34.67 


*** 



Note: ^ The experience of professional development was measured on the scale of 0=no and l=yes. 
^ ns stands for non-signficance, * <.05, ** <.01, and *** <.001. 
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VI. Summary of preliminary Findings and Conclusions: Year 2 oe MSP PD 

Study 

Summary of Results oe Hypothesis Tests 

In this section, we briefly revisit and summarize the results, focusing on the general 
patterns that emerge from these tests of hypotheses. For this purpose, we produced 
summary tables - separately for math (Exhibit 29) and science (Exhibit 30) - which 
allow readers an overview of all the results that were presented in Sections III and IV. 

The summary tables show a number of cells filled in with a probability value of test 
statistics (in this case, based on a set of two-way ANOVAs).*^ The cells are intersections 
between two analytic components. The first component consists of the variables that were 
analyzed for the hypothesis tests. They are listed in rows and organized into four clusters 
(i.e., characteristics of the teacher and target class, instructional activities and strategies, 
the content of classroom instruction, and professional development). The second 
component is made up of six columns which represent hypotheses that were tested or will 
be tested. Under each hypothesis test, we indicated the source of data. For example, 

SECl data were used for the tests of hypotheses 1 and 2, while the PDAL data were used 
for the tests of hypotheses 3 and 4. The follow-up SEC data (SEC2) are scheduled to be 
collected in spring 2005, which will be used to test hypothesis 5. 

In general, the results confirm the expectations set out in hypotheses 1-4. Eirst, as 
expected, in our analysis of hypothesis 1, we found some variation between MSP sites in 
the baseline year. Eor example, both in math and science, the level of teachers’ 
instructional alignment with state standards and assessments differed significantly from 
site to site. The highest degree held by teachers and relational trust among teachers within 
schools was also significantly different between the MSP sites. Also, as expected, in our 
examination of hypothesis 2, we found that the level of teachers’ instructional alignment 
with state standards and assessments did NOT significantly differ between the treatment 
and comparison teachers in the baseline year, although there were some differences 
between mathematics and science teachers. In general, in math, there is very little pre- 
existing difference between the treatment and comparison teachers. In science, however, 
contrary to expectations, the treatment and comparison teachers experienced qualitatively 
different professional development activities in the baseline year. 

As expected, in our analysis of hypothesis 3, we found some difference between the MSP 
sites in the features of professional development teachers experienced in year 2, 
particularly in some structural quality features (e.g., mean contact hours and span). As 
noted in Section IV, we also found some significant difference between the MSP sites in 
their PD activities’ content focus, especially as it relates to topics on instructional 
activities. Consistent with hypothesis 4, we also found differences between treatment and 
comparison teachers in the quality of the professional development experienced during 



The “ns” sign denotes that there is no significant difference between the groups under consideration; * 
sign means the probability (p) of finding the main effect by chance is less than .05; ** p of less than .01; 
and *** p of less than .001. Shaded cells represent not-applicable or non-existent measures. Blank cells 
under SEC2 represent the measures to be analyzed for the Final Report. 
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year 2. The treatment teachers experienced professional development activities that were 
longer in contact hours and span, and focused more on content compared with what their 
comparison counterparts’ professional development activities. 
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Exhibit 29: Summary of Results of Hypothesis Tests: Math 





Comparisons between 


Treatment vs. 


Math 


MSP Program Sites 


Comparison Teachers 




Test of Hypothesis: 


Test of Hypothesis: 




1 3 


2 4 5 


Cluster of Variables 


SEC1' PDAL^ SEC2^ 


sect PDAL^ SEC2^ 



Teacher/Class/School Characteristics 

Course Type 

Number of Students in Target Class 
Achievement Levels of Students(Teacher perception) 

Number of Years in Teaching 
Number of Years in Teaching at This School 
Highest Degree Held 
Relational Trust 

Instructional Activities/Strategies 

Assessment Use 
Instructional Influece 
Cognitive Demands: Memorize 
Cognitive Demands: Perform Procedures 
Cognitive Demands: Communicate Understanding 
Cognitive Demands: Conjecture, Generalize, Prove 
Cognitive Demands: Make Connections 
Content of Classroom Instruction 

Instructional Alignment with Standards 
Instructional Alignment with Assessment 
Cognitive Demands: Memorize 
Cognitive Demands: Perform Procedures 
Cognitive Demands: Communicate Understanding 
Cognitive Demands: Conjecture, Generalize, Prove 
Cognitive Demands: Make Connections 
Breadth of Content Coverage 

Topic Converage: Number Sense/Properties/Relations 
Topic Converage: Operations 
Topic Converage: Measurement 
Topic Converage: Algebraic Concepts 
Topic Converage: Geometric Concepts 
Topic Converage: Data Analysis/Probability/Statistics 
Topic Converage: Instructional Technology 
Professional Development 
Frequency 

% Duration >2-4 days 
Mean Contact Hours 
Activity Span 
Active Learning 
Coherence 

Collective participation (Overall) 

Collective participation (At a given month) 

Strengthen subject matter knowledge 
Number of topical areas focused 
Topic Intensity 

Routine Math Problem-Solving 
Extended Math Problem-solving 
Perform Procedures 
Make Connections 

Number of instructional activities focused 
In-depth study emphasis (single item) 

Content Emphasis 
Data Emphasis 
Student Learning Emphasis 

Standards & Instruction Emphasis 

Note: ^ N of math teachers who responded to SEC1 (Year 1 Survey of Enacted Curriculum) is 210. 

^ N of math teachers who completed the PDAL (Professional Development Activity Log) is 138. 
^ SEC2 measures will be available in spring of 2005. 

Shaded areas represent not-applicable or non-existent measures, 
ns denotes not-significance, * p<.05, ** p<.01 , and *** p<.001 . 
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Exhibit 30: Summary of Results of Hypothesis Tests: Science 





Comparisons between 


Treatment vs. 


Science 


MSP Program Sites 


Comparison Teachers 




Test of Hypothesis: 


Test of Hypothesis: 




1 3 


2 4 5 


Cluster of Variables 


sect PDAL^ SEC2^ 


sect PDAL^ SEC2^ 



Teacher/Class/School Characteristics 

Course Type 

Number of Students in Target Class 

Achievement Levels of Students(Teacher perception) 

Number of Years in Teaching 

Number of Years in Teaching at This School 

Highest Degree Held 

Relational Trust 

Instructionai Activities/Strategies 

Assessment Use 
Instructional Influece 
Cognitive Demands: Memorize 
Cognitive Demands: Perform Procedures 
Cognitive Demands: Communicate Understanding 
Cognitive Demands: Analyze Information 
Cognitive Demands: Make Connections 
Content of Classroom Instruction 

Instructional Alignment with Standards 
Instructional Alignment with Assessment 
Cognitive Demands: Memorize 
Cognitive Demands: Perform Procedures 
Cognitive Demands: Communicate Understanding 
Cognitive Demands: Analyze Information 
Cognitive Demands: Make Connections 
Breadth of Content Coverage 
Topic Converage 
Professional Development 



Frequency 


* 








** 






% Duration >2-4 days 




*** 








* 




Mean Contact Hours 


** 


ns 






*** 


ns 




Activity Span 




*** 








* 




Active Learning 


* 


ns 






** 


ns 




Coherence 


ns 


ns 






** 


ns 




Collective participation (Overall) 


ns 


ns 






ns 


ns 




Collective participation (At a given month) 




ns 








ns 




Strengthen subject matter knowledge 




ns 








*** 




Number of topical areas focused 




ns 








ns 




Topic Intensity 




** 








* 




Laboratory Work 




* 








** 




Science Data/Information Collection 




** 








*** 




Perform Procedures 




* 








* 




Make Connections 




* 








** 




Number of instructional activities focused 




ns 








*** 




In-depth study emphasis (single item) 


ns 








ns 






Content Emphasis 


ns 








* 






Data Emphasis 


ns 








ns 






Student Learning Emphasis 


ns 








ns 






Standards & Instruction Emphasis 


ns 








ns 







Note: ^ N of science teachers who responded to SEC1 (Year 1 Survey of Enacted Curriculum) is 180. 

^ N of science teachers who completed the PDAL (Professional Development Activity Log) is 1 12. 
^ SEC2 measures will be available in spring of 2005. 

"^Some significant effects were found in some of the middle- and high school science topics 
Shaded areas represent not-applicable or non-existent measures, 
ns denotes not-significance, * p<.05, ** p<.01 , and *** p<.001 . 



*** 






*** 






ns 






ns 






ns 






ns 






ns 






ns 






ns 







ns 






ns 






ns 






* 






ns 






* 






* 






ns 






ns 







* 






ns 






* 






ns 






ns 






*** 






* 







ns 






ns 






* 






ns 






ns 






ns 






ns 







*** 






ns 






ns 






ns 






ns 






ns 






* 







ns 






ns 






ns 






* 






ns 






ns 






ns 
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Conclusions: Year 2 of Study 

Testing Effectiveness of Survey Tools in Evaluation. One purpose of the MSP 
PD study is to test the use of teacher self-report survey and web-based tools for 
collecting, analyzing and reporting data on the quality of professional development, and 
the usefulness of these data tools for evaluating effects of professional development. The 
work is being conducted in four MSP sites from the first cohort of MSP grant sites 
(starting Fall 2002) — two comprehensive projects and two targeted projects 

• After the second year of three- year study design, we have demonstrated the use 
of the Surveys of Enacted Curriculum for analyzing differences in instructional practices 
and content of math and science teaching. Teachers across the four MSP sites completed 
the surveys at an overall response rate greater than 80 percent. We demonstrated in the 
Year 2 Progress Report the use of the data for measuring differences in instruction at the 
baseline year, and reporting on differences in the quality and distribution of professional 
development. We mixed a paper-and-pencil survey of teachers and a Web-based 
monthly log with teachers. See edits in ES to make consistent. 

• The study has developed, tested, and demonstrated the use of a web-based 
monthly Professional Development Activity Log (PDAL) system to regularly gather data 
from teachers on the professional development activities. The PDAL system gained 
response rates as high as 62 percent in the 15 months of the test, although the overall 
average rate was 57 percent completing the monthly logs. The study has tested several 
methods of increasing response rates including online and CD training packages, 
postcards, incentive pay, and phone follow-ups. See edits in ES. 

Preliminary Evidence of MSP Supporting Higher Quality Professional 
Development. The study is focusing on a sample of teachers targeted for MSP 
participation and non-participating comparison teachers in each MSP site. Upon 
completing the tests of a series of hypotheses, we found some empirical evidence to 
support the hypothesis that MSP initiatives are achieving their intended purpose of 
supporting quality professional development activities for math and science teachers. For 
example, the four MSP program sites have instituted some of the structural features of 
high quality professional development activities such as extended contact hours and 
prolonged activity span. These are among the structural features likely to set the 
conditions for teachers to receive intense, sustained, and in-depth learning opportunities, 
which, in turn, are expected to produce effects on teaching practice. Furthermore, we also 
found that teachers targeted for participation in MSP programs tend to experience a 
stronger content focus in their professional development, particularly in science. 
Combined with sustained learning opportunities, the stronger content focus afforded by 
MSP programs is expected to allow teachers to practice new content and instructional 
strategies they received in their classrooms. 

When the follow-up SEC data are made available in spring of 2005, we will test the most 
interesting and important hypothesis of this study. At the outset of this report, we 
advanced Hypothesis 5, which states, in part, “after a year of MSP implementation, we 
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expect teachers in the treatment group (i.e., teachers targeted for MSP participation) to 
exhibit higher quality instruction, as measured by the alignment of instruction with state 
content standards, than teachers in the comparison group, after controlling for any year 1 
differences.” Our analysis of the effects of professional development is based on a quasi- 
experimental analysis of teachers targeted for participation in MSP (for simplicity termed 
“treatment” teachers), and a group of non-participating comparison teachers. While 
random assignment of teachers to MSP and comparison groups would be ideal, this was 
not feasible in the current study. Thus, in order to assess the effect of teachers’ 
professional development experiences on their instructional practice, it is essential to 
control for any differences found to exist at the baseline year between MSP sites as well 
as between “treatment” and comparison teachers. We have collected extensive baseline 
data which can be included as potential control variables. Including these variables in the 
analysis will strengthen the attribution of any improvement in instruction among the 
treatment teachers to the professional development experiences they had experienced 
over the period of the MSP program. Make changes to be consistent with edits in ES. 

This test of the effects of PD on is made possible through our longitudinal study design 
and the use of such innovative instruments as the PDAL and SEC funded by NSF’s 
RETA grant program. Make consistent with ES. 
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Appendix A - Teacher Responses to Surveys of Enacted Curriculum Year 1 (2003) and 

Professional Development Activity Log (July 2003-Sept. 2004), MSP PD Study 

Intended 

Sample Year 1 SEC PDAL 

Response Completion 







Math 


Science 


Total 


Rate 


Math 


Science 


Total 


Rate 


MSP Site 




















Brockport 


91 


47 


31 


78 


86% 


37 


18 


55 


60% 


Cleveland 


180 


79 


99 


178 


99% 


49 


63 


112 


62% 


Corpus Christ! 


93 


50 


28 


78 


84% 


34 


23 


57 


61% 


El Paso 


112 


34 


22 


56 


50% 


28 


21 


49 


44% 


Treatment Status 




















Comparison 


213 


77 


92 


169 


79% 


45 


54 


99 


46% 


Treatment 


263 


133 


88 


221 


84% 


103 


71 


174 


66% 


MSP_Site*Treatment Status 




















Brockport-Comparison 


43 


19 


17 


36 


84% 


12 


8 


20 


47% 


Brockport-T reatment 


48 


28 


14 


42 


88% 


25 


10 


35 


73% 


Cleveland-Comparison 


68 


28 


40 


68 


100% 


14 


15 


29 


43% 


Cleveland-T reatment 


112 


51 


59 


110 


98% 


35 


48 


83 


74% 


Corpus Christi-Comparison 


51 


15 


26 


41 


80% 


6 


22 


28 


55% 


Corpus Christi-T reatment 


42 


35 


2 


37 


88% 


28 


1 


29 


69% 


El Paso-Comparison 


51 


15 


9 


24 


47% 


13 


9 


22 


43% 


El Paso-Treatment 


61 


19 


13 


32 


52% 


15 


12 


27 


44% 


Total 


476 


210 


180 


390 


82% 


148 


125 


273 


57% 



Note: ' PDAL completion rate represents the number of teachers who completed usable PDAL data among the intended 
sample. 
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Appendix B - Mathematics Scales 

Reliability Coefficient 



Assessment Use 0.727 

Q64 Short answer questions such as performing a mathematicai 0.71 3 

Q65 Extended response item for which student must expiain or justify 0.663 

Q66 Performance tasks or events (e.g. hands-on activities). 0.669 

Q67 Individual or group demonstration, presentation. 0.671 

Q68 Mathematics projects. 0.693 

Q69 Portfolios. 0.726 

Q70 Systematic observation of students. 0.730 

Influence of Standards 0.674 

Q71 Your state's curriculum framework or content standards. 0.543 

Q72 Your district's curriculum framework or guidelines. 0.580 

Q76 National mathematics education standards. 0.700 

Q83 Provide mathematics instruction that meets mathematics content standards (district, 0.693 
Q128 State mathematics content standards (e.g. what they are and how they are used). 0.609 
Q129 Alignment of mathematics instruction to curriculum. 0.637 

Climate of Trust 0.823 

Q93 I am supported by colleagues to try out new ideas in teaching mathematics. 0.81 5 

Q96 Mathematics teachers in this school trust each other. 0.784 

Q97 It's OK in this school to discuss feelings, worries, and frustrations with other ^ 

mathematics teachers. 

Q98 Mathematics teachers respect other teachers who take the lead in school 
improvement efforts. 

Q99 It's OK in this school to discuss feelings, worries, and frustrations with the principal. ^ 



01 00 The principal takes personal interest in the professional development of the teachers. 



Perform Procedures 0.758 

Q37 Solve word problems from a textbook or worksheet. 0.760 

045 Solve word problems from a textbook or worksheet. 0.745 

Q53* Work with manipulatives (e.g. counting blocks, geometric shapes, or algebraic tiles) to ^ 
understand concepts. 

Q54* Measure objects using tools such as rulers, scales, or protractors. 0.701 

Q56* Collect data by counting, observing, or conducting surveys. 0.687 

Q59 Practice procedures 0.745 

Q60B Retrieve or exchange data or information (e.g. using the Internet or partnering with ^ 
another class) 

Communicative Understanding 0.802 

Q29 Present or demonstrate solutions to a math problem to the whole class. 0.787 

Q32* Work in pairs or small groups on math exercises, problems, investigations, or tasks. _ 



Q39 Explain their reasoning or thinking in solving a problem, using several sentences orally „ 

U./DO 

or in writing. 

Q47 Talk about their reasoning or thinking in solving a problem. 0.749 

Q57 Present information to others using manipulatives (e.g. chalkboard, whiteboard, ^ 

posterboard, projector). 



Note: Results for individual items in Reliability Coefficient column report coefficient if item is deleted. 
* Item used in multiple scales (for exploratory purposes only). 



MEAN 

1.786 

2.590 

2.240 

1.977 

1.392 

0.954 

0.857 

2.498 

3.059 

4.217 

4.272 

3.622 

2.185 

1.968 
2.083 

2.792 

2.903 

2.727 

2.958 

2.968 
2.579 
2.620 



2.307 

2.806 

2.565 

2.318 

2.493 

2.014 

2.815 

1.140 



2.506 

2.542 

2.880 

2.083 

2.569 

2.458 
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Appendix B - Mathematics Scales (continued) 



Reliability Coefficient 



Analyze Information (Conjectures, Generalize, Prove Math) 0.868 

Q41 Make estimates, predictions or hypotheses. 0.847 

Q42 Anaiyze data to make interferences or draw conclusions. 0.837 

Q44 Complete or conduct proofs or demonstrations of their mathematical reasoning. 0.849 

Q49 Make estimates, predictions or hypotheses. 0.843 

Q52 Complete or conduct proofs or demonstrations of their mathematical reasoning. 0.822 



Make Connections (Solve new notions) 

Q38 Solve non-routine mathematical problems (e.g. problems that require novel or non- 
formulaic thinking). 

Q40 Apply mathematical concepts to "real-world" problems. 

Q46 Solve non-routine mathematical problems (e.g. problems that require novel or non- 
formulaic thinking). 

Q48 Apply mathematical concepts to "real-world" problems. 

Q50 Apply data to make inferences or draw conclusions. 

Q51 Work on a problem that takes at least 45 minutes to solve. 



0.861 

0.857 

0.845 

0.823 

0.819 

0.823 

0.854 



Active Learning 0.853 

Q30 Use manipulatives (e.g. counting blocks, geometric shapes, or algebraic tiles), 

measurement instruments (e.g. rulers or protractors), and data collection devices (e.g. o.805 

surveys or probes). 

Q32* Work in pairs or small groups on math exercises, problems, investigations, or tasks. ^ 



Q33 Do a mathematics actively with the class outside the classroom. 0.854 

Q53* Work with manipulatives (e.g. counting blocks, geometric shapes, or algebraic tiles) to q 
understand concepts. 

Q54* Measure objects using tools such as rulers, scales, or protractors. 0.81 6 

Q56* Collect data by counting, observing, or conducting surveys. 0.81 6 

PD Frequency For the most recent school year, how often have you participated in: 0.351 

q1 01 a Workshops or in-service training related to mathematics or mathematics education 0.452 

q102a Summer institutes related to mathematics or mathematics education 0.179 

q103a College courses related to mathematics or mathematics education 0.245 



PD Hours 



For the most recent school year, how many total hours have you 
participated in: 

qlOlb Workshops or in-service training related to mathematics or mathematics education 
q102b Summer institutes related to mathematics or mathematics education 
q103b College courses related to mathematics or mathematics education 



0.461 

0.460 

0.066 

0.504 



Active Teacher Engagement PD 0.767 

q111 Observed demonstrations of teaching techniques 0.743 

q1 12 Led group discussions. 0.723 

q1 1 3 Developed curricula or lesson plans, which other participants or the activity leader 

, U. / 

reviewed. 

q1 14 Reviewed student work or scored assessments. 0.777 



q1 1 5 Developed assessments or tasks as part of a formal professional development activity. ^ ^^5 

q1 1 6 Practiced what you learned and received feedback as part of a professional ^ 

development activity. 

q1 1 7 Received coaching or mentoring in the classroom. 0.755 

q1 1 8 Given a lecture or presentation to colleagues. 0.751 

Note: Results for individual items in Reliability Coefficient column report coefficient if item is deleted. 

* Item used in multiple scales (for exploratory purposes only). 



MEAN 

1.957 

2.349 

2.293 

1.403 

2.241 

1.495 

2.164 

1.944 

2.875 

1.972 

2.763 

2.265 

1.167 

2.217 

2.677 



2.880 

0.917 

2.318 

2.493 

2.014 

2.342 

5.337 

0.853 

0.836 



16.264 

31.099 

10.533 

7.161 

1.145 

1.442 

0.921 

1.321 

2.030 

1.036 

1.279 

0.612 

0.521 



68 



Appendix B - Mathematics Scales (continued) 



Reliability Coefficient 



Coherent PD Program 0.752 

q1 1 9 Designed to support the school-wide improvement plan adopted by your school. 0.741 

q120 Consistent with you mathematics department or grade level plan to improve teaching. ^ 

q1 21 Consistent with your own goals for your professional development. 0.686 



q122 Based explicitly on what you had learned in earlier professional development activities. 



q123 Followed up with related activities that built upon what you learned as part of the 
activity 



0.688 



Collective Participation (sum) 0.677 

q124 I participated in professional development activities with most or all of the teachers 
from my school. 

q125 I participated in professional development activities with most or all of the teachers 
from my department or grade level. 



PD w/ Content Focus 0.746 

q1 28* State mathematics content standards (e.g. what they are and how they are used). 0.687 
q1 29* Alignment of mathematics instruction to curriculum. 0.658 

q1 31 * In-depth study of mathematics or specific concepts within mathematics ^ 

(e.g. fractions). 

q1 32* Study of how children learn particular topics in mathematics. 0.730 

PD w/ Data Focus 0.824 



q135* Classroom mathematics assessment (e.g. diagnostic approaches, textbook-developed ^ 
tests, teacher-developed tests). 

q1 36* State or district mathematics assessment (e.g. preparing for assessments, g ^^4 

understanding assessments, or interpreting assessments). 
q1 37* Interpretation of assessment data for use in mathematics instruction. 0.704 



PD w/ Standards & Instruction Focus 

q128* State mathematics content standards 

(e.g. what they are and how they are used). 
q129* Alignment of mathematics instruction to curriculum. 
q131* In-depth study of mathematics or specific concepts within mathematics 
(e.g. fractions). 

q136* State or district mathematics assessment (e.g. preparing for assessments, 
understanding assessments, or interpreting assessments). 
q137* Interpretation of assessment data for use in mathematics instruction. 



0.830 

0.804 

0.795 

0.809 

0.787 

0.789 



PD w/ Student Learning Focus 0.818 

q1 32* Study of how children learn particular topics in mathematics. 0.750 

q133 Individual differences in student learning. 0.736 

q134 Meeting the learning needs of special populations of students g 

(e.g. second language learners; students with disabilities). 
q135* Classroom mathematics assessment (e.g. diagnostic approaches, textbook-developed g gg^ 
tests, teacher-developed tests). 

q138 Technology to support student learning in mathematics. 0.836 



Note: Results for individual items in Reliability Coefficient column report coefficient if item is deleted. 
* Item used in multiple scales (for exploratory purposes only). 



MEAN 

1.878 

1.788 

2.111 

2.174 

1.623 

1.693 

0.652 

0.558 

0.745 

1.750 

2.079 

2.177 

1.604 

1.141 

1.482 

1.313 

1.663 

1.470 

1.689 

2.079 

2.177 

1.604 

1.663 

1.470 

1.236 

1.141 

1.258 

1.049 

1.313 

1.421 
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Appendix B - Science Scales 



Reliability Coefficient 



Assessment Use 0.743 

Q64 Short answer questions (e.g. fill-in-the-blank). 0.735 

Q65 Extended response item for which student must explain or justify solution. 0.689 

Q66 Performance tasks or events (e.g. hands-on activities). 0.680 

Q67 Individual or group demonstration, presentation. 0.694 

Q68 Science projects. 0.721 

Q69 Portfolios. 0.729 

Q70 Systematic observation of students. 0.736 



Influence of Standards 

Q71 Your state's curriculum framework or content standards. 

Q72 Your district's curriculum framework or guidelines. 

Q76 National science education standards. 

Q83 Provide science instruction that meets science content standards (district, state, or 
national). 

Q128 State science content standards (e.g. what they are and how they are used). 

Q129 Alignment of science instruction to curriculum. 



0.761 

0.685 

0.719 

0.725 

0.764 

0.735 

0.723 



Climate of Trust 

Q93 I am supported by colleagues to try out new ideas in teaching science. 

Q96 Science teachers in this school trust each other. 

Q97 It's OK in this school to discuss feelings, worries, and frustrations with other science 
teachers. 

Q98 Science teachers respect other teachers who take the lead in school improvement 
efforts. 

Q99 It's OK in this school to discuss feelings, worries, and frustrations with the principal. 
The principal takes personal interest in the professional development of the teachers. 



0.817 

0.777 

0.779 

0.775 

0.780 

0.808 

0.808 



Perform Procedures 0.881 

029 Do a laboratory activity, investigation, or experiment. 0.859 

038 Follow step-by-step directions. 0.866 

Q39* Use science equipment or measuring tools. 0.852 

040 Collect data. 0.848 

042 Organize and display information in tables or graphs. 0.853 

045 Make observations/classifications. 0.857 

058 Practice procedures. 0.890 

059* Use sensors and probes (e.g. Computer Based Labs) 0.894 



Communicative Understanding 

028 Write about science in a report/paper on science topics. 

046 Complete written assignments from the textbook or workbook. 

048 Write up results or prepare a presentation from a laboratory activity, investigation, 
experiment or a research project. 

050 Work on a writing project or entries for portfolios seeking paper comments to improve 
work. 

052 Have class discussions about the data. 

053 Organize and display the information in tables or graphs. 

056 Make a presentations to the class on the data, analysis, or interpretation. 



0.884 

0.878 

0.873 

0.866 

0.865 

0.868 

0.864 

0.858 



Note: Results for individual items in Reliability Coefficient column report coefficient if item is deleted. 
* Item used in multiple scales (for exploratory purposes only). 



MEAN 

1.882 

2.306 

2.210 

2.344 

1.613 

1.215 

1.156 

2.333 

3.011 

4.290 

4.172 

3.597 

2.134 

1.903 

1.968 

2.706 

2.887 

2.726 

2.790 

2.844 

2.468 

2.522 



2.715 

2.968 

3.134 

3.145 

3.177 

2.968 

3.247 

2.366 

0.715 

2.060 

1.925 

2.258 

2.430 

1.215 

2.349 

2.328 

1.914 
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Appendix B - Science Scales (continued) 

Anaiyze Information 

Q43 Analyze and interpret science data. 

Q54 Make a prediction based on the data. 

Q55 Analyze and interpret the information or data, orally or in writing. 
Q61 Display and analyze data. 



Reliability Coefficient 
0.834 

0.810 

0.742 

0.752 

0.852 



Make Connections 0.809 

Q37 Make educated guesses, predictions, or hypotheses. 0.773 

Q41 Collect data. 0.692 

Q44* Design their own investigation or experiment to solve a scientific question. 0.749 



Active Learning 0.833 

Q29 Do a laboratory activity, investigation, or experiment. 0.789 

Q31 Collect data (other than laboratory activities). 0.801 

Q34* Use computers, calculators or other educational technology or learn science. 0.816 

Q39* Use science equipment or measuring tools. 0.805 

Q44 Design their own investigation or experiment to solve a scientific question. 0.791 

Q59* Use sensors and probes (e.g. Computer Based Labs). 0.834 

PD Frequency For the most recent school year, how often have you participated in: 0.552 

q1 01 a workshops or in-service training related to science or science education 0.483 

q102a summer institutes related to science or science education 0.242 

q1 03a college courses related to science or science education 0.606 



PD Hours 



For the most recent school year, how many total hours have you 
participated in: 

qlOlb workshops or in-service training related to science or science education 
q102b summer institutes related to science or science education 
q103b college courses related to science or science education 



0.502 

0.293 

0.154 

0.648 



Active Teacher Engagement PD 

q1 1 1 Observed demonstrations of teaching techniques. 
q112 Led group discussions. 

q1 13 Developed curricula or lesson plans, which other participants or the activity leader 
reviewed. 

q1 14 Reviewed student work or scored assessments. 

q1 15 Developed assessments or tasks as part of a formal professional development activity. 

q1 16 Practiced what you learned and received feedback as part of a professional 
development activity. 

q1 17 Received coaching or mentoring in the classroom. 
q1 18 Given a lecture or presentation to colleagues. 

Coherent PD Program 

q1 19 Designed to support the school-wide improvement plan adopted by your school. 
q120 Consistent with you science department or grade level plan to improve teaching. 
q121 Consistent with your own goals for your professional development. 
q122 Based explicitly on what you had learned in earlier professional development activities. 



0.830 

0.811 

0.794 

0.805 

0.825 

0.799 

0.807 

0.828 

0.812 

0.855 

0.852 

0.807 

0.797 



0.835 



q123 Followed up with related activities that built upon what you learned as part of the 
activity 

Note: Results for individual items in Reliability Coefficient column report coefficient if item is deleted. 
* Item used in multiple scales (for exploratory purposes only). 



0.835 



MEAN 

2.470 

3.016 

2.290 

2.430 

2.145 

2.385 

2.882 

2.392 

1.882 

2.159 

2.968 

1.995 

2.253 

3.145 
1.882 
0.715 

2.424 

4.612 

1.281 

1.380 

14.083 

20.273 
1 1 .843 
10.132 

1.219 

1.387 

0.927 

1.439 

1.610 

1.734 

1.460 

0.540 

0.653 

1.769 

1.606 

2.018 

2.036 

1.500 

1.685 
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Appendix B - Science Scales (continued) 



Reiiability Coefficient 

Coiiective Participation (sum) 0.756 

q124 I participated in professional development activities with most or all of the teachers 
from my school. 

q125 I participated in professional development activities with most or all of the teachers 
from my department or grade level. 



PD w/ Content Focus 0.839 

q128* State science content standards (e.g. what they are and how they are used). 0.788 

q129* Alignment of science instruction to curriculum. 0.781 

q131* In-depth study of science or specific concepts within science (e.g. earth science). . 

U. lod. 

q132* Study of how children learn particular topics in science. 0.830 

PD w/ Data Focus 0.826 



q135* Classroom science assessment (e.g. diagnostic approaches, textbook-developed 
tests, teacher-developed tests). 

q136* State or district science assessment (e.g. preparing for assessments, understanding 
assessments, or interpreting assessments). 



q1 37* Interpretation of assessment data for use in science instruction. 0.712 

PD w/ Standards & Instruction Focus 0.867 

q128* State science content standards (e.g. what they are and how they are used). 0.858 

q129* Alignment of science instruction to curriculum. 0.861 

q131* In-depth study of science or specific concepts within science (e.g. earth science). 0.858 

q136* State or district science assessment (e.g. preparing for assessments, understanding ^ 

assessments, or interpreting assessments). 

q137* Interpretation of assessment data for use in science instruction. 0.856 



PD w/ Student Learning Focus 

q132* Study of how children learn particular topics in science. 
q133 Individual differences in student learning. 

q134 Meeting the learning needs of special populations of students (e.g. second language 
learners; students with disabilities). 

q135* Classroom science assessment (e.g. diagnostic approaches, textbook-developed 
tests, teacher-developed tests). 
q138 Technology to support student learning in science. 



0.865 

0.839 

0.813 

0.826 

0.847 

0.858 



Note: Results for individual items in Reliability Coefficient column report coefficient if item is deleted. 
* Item used in multiple scales (for exploratory purposes only). 



MEAN 

0.508 

0.444 

0.573 

1.676 

1.942 

2.074 

1.512 

1.174 

1.311 

1.298 

1.430 

1.207 

1.556 

1.942 

2.074 

1.512 

1.430 

1.207 

1.261 

1.174 

1.298 

1.198 

1.298 

1.339 
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Major Types of Teacher Professional Development in Mathematics Education Activities 
Reported by Teachers During Past 12 Months (SEC, Year 1) 



Grade 



□ Middle School i — i High School 
(N=166) I I (N=50) 



PM^ssionaJ 



Percent of Math Teachers 



Workshops or in-service training related to 
mathematics or mathematics education 




Summer Institutes related to mathematics 
and mathematics education 




College courses related to mathematics or 
mathematics education 




Source: Survey of Enacted Curriculum, MSP Study: Year One Teacher Surveys, Spring 2003 
CCSSO, AIR, WCER, National Science Foundation, MSP-RETA Grant 



Major Types of Teacher Professional Development in Science Education Activities Reported 
by Teachers During Past 12 Months (SEC, Year 1) 



Grade 



□ Middle School i 1 High School 

(N = 121) I I (N=60) 



Professional Development in Science 



Percent of Science Teachers 



Workshops or in-service training related to 
science or science education 




Summer Institutes related to science or 
science education 




College courses related to science or 
science education 




Source: Survey of Enacted Curriculum, MSP Study: Year One Teacher Surveys, Spring 2003 
CCSSO, AIR, WCER, National Science Foundation, MSP-RETA Grant 



Active Learning of Professional Development Activities in Middle School Mathematics and 
Science 



Frequency 

I I Never Sometimes 
m Rarely Q Often 



Degree of Active Learning 



Percent of Teachers 

Mathematics (N=1 66) Science (N=1 21 ) 



Observed demonstrations of teaching 

techniques 



Led group discussions 



D 2b% 






Practiced what you learned or received feedback 



Received coaching or mentoring in the 

classroom 



Gave a lecture or presentation to colleagues 



I 

0 32 % 



5Wii 



L 



JI3 

I I 

1 00% 0 24% 



:□ 



100 % 



I I 
100 % 0 



44% 



100 % 



Developed curricula or lesson plans, which 


E 


— ^ — 


1 1 


other participants or the activity leader reviewed 


1 




1 




0 


32^ P 


100% 

.j.. 


Reviewed student work or scored assessments 


1 


M 1 ~ 


1 

1 




1 

D 


12% 


1 

10D% 


Developed assessments or tasks 


c 

1 




— n 

1 




1 

0 


41% 


1 

100% 



L 



n I 



I 

100 % 



100 % 



I I 



100 % 



1 — I 



I I 
100 % 0 



100 % 



I I 
100 % 0 



I 

03 % 100 % 



I I 
100% 0 



00% 



I 

100 % 



Source: Survey of Enacted Curriculum, MSP Study: Year One Teacher Surveys, Spring 2003 
CCSSO, AIR, WCER, National Science Foundation, MSP-RETA Grant 



Coherence of Professional Development Activities in Middle School Mathematics and 
Science 



Frequency 

I I Ne^ ^ Purely Sometimes 
[] Often I N/A 



Coherence of Professional Development 



Designed to support the school-wide 
improvement plan adopted by your school ’ 



Percent of Teachers 

Mathematics (N=1 66) Science (N=1 21 ) 



0 14^0 



Consistent with your department or grade 



0 



Consistent with your own goals for your 
professional development 

Based explicitly on what you had learned in 
earlier professional development activities I 



0 R% 



Followed up with related activities that built upon 
what you learned as part of the activity 






I 







1 

0 22'Kn 


1 

100% 


M 1 


1 

0 12<>o 


1 

100%. 


1 

1 


1 

0 


1 

lfK)% 




1 

0 17% 


1 





I I 

100 % 0 16 % 



I 

100 % 



Source: Survey of Enacted Curriculum, MSP Study Yar One Teacher Surveys, Spring 2003 
CCSSO, AIR, WCER, National Science Foundation, MSP-RETA Grant 



Collegial Participation in Professional Development Activities in Middle School 
Mathematics and Science 



Collegial Participation 



Percent of Teachers 

Mathematics (N=1 66) Science (N=1 21 ) 



Participated in professional development 


KVj 

1 1 


YCh! 


activities with most or all of the teachers from 


1 II 

1 


1 


you school 


u 




Participated in professional development 


Kn 


Yc-fi 


activities with most or all of the teachers from 


1 


1 


your department or school level 


U 


1 00 


Participated in professional development 


Kn 


Yc-fi 


activities not attended by other staff member 


1 

1 


1 


from your school 


a 


1 



Discussed what was learned with other Kpi_ 
teachers in your school or department who did | 
not attend the activity () 



Ycfi 



No 


Yc-s 


1 


1 

0 


1 

100°^“ 


No 


Yes 


1 


1 

0 


1 

1 00 


No 


Y"Cf! 


1 


1 

0 


1 

loo'^^ 


No 


Yes 



IhO" 



1 DO ^ 



Source: Survey of Enacted Curriculum, MSP Study: Year One Teacher Surveys, Spring 2003 
CCSSO, AIR, WCER, National Science Foundation, MSP-RETA Grant 



Content Focus of Professional Development Activities in Middle School Mathematics and 
Science 



Emphasis 

I I None I I MnHpratP 
m Slight 1^ Great 



Topic of Professional Development 



State content standards 



Alignment of instruction to curriculum 



Technology to support student learning 



Percent of Teachers 

Mathematics (N=1 66) Science (N=1 21 ) 

I ~~ 



0 7" 



0 



Instructional approaches 



In-depth study of subject or specific concepts 



Study of how children learn particular topics 



Individual differences in student learning 



Meeting the learning needs of special 
populations of students 



Classroom assessment 



State and district assessment 



0 10% 



I 

0 17"i. 



31% 



0 27% 



S9"n 



1% 



0 13% 



Interpretation of assessment data for use in I^^IM 

instruction I 
0 



0 20% 



U I IM 

I I 

L00% 0 10->n 



U 
I I 

100% 0 l(^^n 



1 I 



I I 

100 % 0 10 % 



I I 

100 % 1 ) 26 “ 0 



JU C 

I I 

L00% 0 






HD 



I I 

100% 0 20^,) 



HD 



E 



I I 

100 %. 0 



n'% 



E 



I I 

1 00% 0 2h%> 



:□ I iM 

I I 

100% 0 31^% 



HDi 



I I 

100 % 0 20 %, 



X 



100 % 



100 % 



I 

100 % 



"1 1 



100 % 



HD 

I 

100%, 



HD 

I 

100 %, 



HD 

I 

100%, 



HD 



I 

100 %, 



1 — I 



100 % 



HD 



100 %, 



HZi 



I 

100 %^ 



Source: Survey of Enacted Curriculum, MSP Study: Year One Teacher Surveys, Spring 2003 
CCSSO, AIR, WCER, National Science Foundation, MSP-RETA Grant 



Types of Professional Development Activities Specific to Teaching and Learning Middle 
School Mathematics and Science 



Frequency 



I I Newer 0 Once or twice a term | Once or twice a week 

H Once or twice a year Once or twice a month Almost daily 



Types of Activities 



Percent of Teachers 

Mathematics (N=1 66) Science (N=1 21 ) 



Attended conferences related to subject q 

or education | 

0 



Participated in teacher study group 



Participated in a teacher network, or □ 
collaborative of teachers supporting I 
professional development q 

Acted as a coach or mentor to other teachers 
or staff in our school 



Received coaching or mentoring 



Participated in a committee or task force 



Informal self-directed learning e.g. discussion 
with colleagues, read a journal article 



42^' r 



54% 



37\ 



S2% 



C 

I I 

ior>% i) 



n — B i 



I I 

1 00% 0 



I I 
im% 0 



I I 
100 % 0 



Tlff l 



13 



45% 



100 % 



XDD 



I 

GPo 100% 

I III 



44% 



I 

100 % 



"nan 



52% 



I 

100 % 



1 

0 


SWi, 


1 

100% 


1 

0 




1 

100% 


1 




1 II 


1 




■ 


1 

0 


52% 


1 

100% 


1 

0 


47% 


1 

100% 


1 

■ 

1 

' 


1 

0 10% 




1 

100% 


1 

0 12^^ 




1 

100% 



Source: Survey of Enacted Curriculum, MSP Study: Year One Teacher Surveys, Spring 2003 
CCSSO, AIR, WCER, National Science Foundation, MSP-RETA Grant 



Appendix D1. Descriptive Statistics for PDAL Variabies: Based on Monthiy Log Levei Data (N=1,797) 



Variable 


Description 


N 


Mean 


SD 


Min 


Max 


Activity Duration, Contact Hours, and Span 


Act_Du ration 


How many days: On the scale of 0=less than a day, 1 =one day, 2=2-4 days, 3=a 
week, 4=entire month. 


1797 


1.87 


1.30 


0.0 


4.0 


ActD_Du ration 


Activity duration dummy variable: 0=one day or less, 1= 2-4 days or longer 


1797 


0.58 


0.49 


0.0 


1.0 


Act_Hours 


Activity Contact Hour 


1797 


15.37 


19.22 


1.0 


160.0 


Act_Hour307 


Activity Contact Hour during July 2003 


206 


35.82 


28.72 


1.0 


150.0 


Act_Hour308 


Activity Contact Hour during August 2003 


215 


13.84 


15.34 


1.0 


80.0 


Act_Hour309 


Activity Contact Hour during September 2003 


152 


13.07 


18.23 


1.0 


160.0 


Act_Hour310 


Activity Contact Hour during October 2003 


216 


11.01 


14.94 


1.0 


160.0 


Act_Hour31 1 


Activity Contact Hour during November 2003 


148 


11.52 


17.20 


1.0 


160.0 


Act_Hour312 


Activity Contact Hour during December 2003 


130 


9.35 


14.34 


1.0 


140.0 


Act_Hour401 


Activity Contact Hour during January 2004 


128 


8.71 


15.44 


1.0 


160.0 


Act_Hour402 


Activity Contact Hour during February 2004 


140 


10.74 


12.28 


1.0 


80.0 


Act_Hour403 


Activity Contact Hour during March 2004 


130 


10.36 


10.58 


1.0 


60.0 


Act_Hour404 


Activity Contact Hour during April 2004 


88 


11.66 


12.48 


1.0 


70.0 


Act_Hour405 


Activity Contact Hour during May 2004 


96 


11.77 


9.99 


1.0 


60.0 


Act_Hour406 


Activity Contact Hour during June 2004 


58 


25.26 


20.78 


1.0 


80.0 


Act_Hour407 


Activity Contact Hour during July 2004 


60 


29.29 


19.65 


2.5 


80.0 


Act_Hour408 


Activity Contact Hour during August 2004 


30 


20.97 


16.87 


1.0 


60.0 


Act_Continued 


Activity is continued (0=no, 1 =yes) 


1797 


0.69 


0.46 


0.0 


1.0 


Act_Mo_N 


Activity months: Count of active months only 


1797 


3.04 


2.59 


1.0 


12.0 


Act_Span 


Activity Span: Count of months from start to the end (inclusive months) 


1797 


3.95 


3.54 


1.0 


14.0 


Act_Consecutive 


Activity occurred in consecutive months (0=no, 1=yes) 


1295 


0.55 


0.50 


0.0 


1.0 


Activity Type 


Dummy variables (0=no, 1=yes) 












Act_workshop 


PD Activity Type - Participation in a workshop or in-service activity 


1021 


0.58 


0.49 


0.0 


1.0 


Actjnst 


PD Activity Type - Participation in a summer institute 


1021 


0.12 


0.32 


0.0 


1.0 


Act_college 


PD Activity Type - Attendance at a college course 


1021 


0.16 


0.37 


0.0 


1.0 


Act_confer 


PD Activity Type - Attendance at a conference 


1021 


0.08 


0.27 


0.0 


1.0 


Act_studygr 


PD Activity Type - Participation in a teacher study group 


1021 


0.16 


0.36 


0.0 


1.0 


Act_network 


PD Activity Type - Participation in a teacher network or collaborative of teachers 


1021 


0.27 


0.44 


0.0 


1.0 


Act_mentor 


PD Activity Type - Working with a mentor, coach, lead teacher, or observer 


1021 


0.17 


0.38 


0.0 


1.0 


Act_taskforce 


PD Activity Type - Participation in a teacher committee or task force 


1021 


0.10 


0.29 


0.0 


1.0 


Actjnformal 


PD Activity Type - Engagement in informal self-directed learning 


1021 


0.16 


0.37 


0.0 


1.0 


PD Quaiity Scaies 


ActiveLearn 


Active Learning (mean score): 0=never, 1=rarely, 2=sometimes, 3=often 


1781 


1.10 


0.76 


0.0 


3.0 


Coherence 


Coherence (mean score): 0=never, 1=rarely, 2=sometimes, 3=often 


1709 


2.32 


0.67 


0.0 


3.0 


Collective 


Collective Participation (sum score): On the scale of 0 - 2: number of postive 
response to two collective participation items 


1766 


0.62 


0.73 


0.0 


2.0 
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Appendix D1. Descriptive Statistics for PDAL Variabies: Based on Monthiy Log Levei Data (N=1,797) 



Variable 


Description 


N 


Mean 


SD 


Min 


Max 


Content Focus Measures 


Purpose of PD 


Purp_Subject 


Strengthening subject matter knowledge 


1689 


0.71 


0.46 


0.0 


1.0 


Math Topics 


topics_math 


Total number of 1 st-order topics covered in the activity 


990 


1.95 


2.10 


0.0 


6.0 


topic_number 


Number of Sub-topics Covered - Number sense/Properties/Relationships 


989 


3.28 


5.26 


0.0 


16.0 


topic_measure 


Number of Sub-topics Covered - Measurement 


990 


2.44 


4.61 


0.0 


15.0 


topic_algebra 


Number of Sub-topics Covered - Algebraic Concepts 


990 


5.53 


7.50 


0.0 


21.0 


topic_geometry 


Number of Sub-topics Covered - Geometric Concepts 


990 


3.15 


5.52 


0.0 


17.0 


topic_operation 


Number of Sub-topics Covered - Operations 


989 


2.95 


5.38 


0.0 


16.0 


topic_analysis 


Number of Sub-topics Covered - Data Analysis/Probability/Statistics 


989 


3.42 


5.30 


0.0 


15.0 


cover_number 


Math Content Covered - Number Sense/Properties/Relationships 


989 


0.35 


0.48 


0.0 


1.0 


cover_measure 


Math Content Covered - Measurement 


990 


0.27 


0.44 


0.0 


1.0 


cover_algebra 


Math Content Covered - Algebraic Concepts 


990 


0.40 


0.49 


0.0 


1.0 


cover_geometry 


Math Content Covered - Geometric Concepts 


990 


0.31 


0.46 


0.0 


1.0 


cover_operation 


Math Content Covered - Operations 


989 


0.28 


0.45 


0.0 


1.0 


cover_analysis 


Math Content Covered - Data Analysis/Probability/Statistics 


989 


0.35 


0.48 


0.0 


1.0 


intensetopic_math 


Math topic intensity (contact hours divided by number of 1 st-order topics) 


990 


4.74 


8.72 


0.0 


80.0 


Science Topics 


topics_science 


Total number of 1 st-order topics covered in the activity 


805 


4.22 


6.30 


0.0 


25.0 


topic_nature_sc 


Number of sub-topics covered - Nature of Science 


805 


1.20 


1.70 


0.0 


5.0 


topic_sc_tech 


Number of sub-topics covered - Science and Technology 


805 


0.77 


1.13 


0.0 


3.0 


topic_health_env 


Number of sub-topics covered - Science, Health and Environment 


805 


0.72 


1.38 


0.0 


4.0 


topic_measure_cal 


Number of sub-topics covered - Measurement and Calculation in Science 


805 


2.59 


4.21 


0.0 


12.0 


topic_compo_live 


Number of sub-topics covered - Components of Living Systems 


804 


1.13 


2.47 


0.0 


7.0 


topic_botany 


Number of sub-topics covered - Botany 


804 


0.66 


1.76 


0.0 


6.0 


topic_animal_bio 


Number of sub-topics covered - Animal Biology 


804 


0.97 


2.50 


0.0 


8.0 


topic_human_bio 


Number of sub-topics covered - Human Biology 


804 


0.93 


2.46 


0.0 


8.0 


topic_ecology 


Number of sub-topics covered - Ecology 


804 


0.71 


1.80 


0.0 


6.0 


topic_energy 


Number of sub-topics covered - Energy 


803 


1.09 


2.31 


0.0 


7.0 


topic_motion_force 


Number of sub-topics covered - Motion and Forces 


803 


1.03 


2.49 


0.0 


8.0 


topic_waves 


Number of sub-topics covered - Characteristics and Behavior of Waves 


803 


0.33 


0.89 


0.0 


3.0 


topic_kinetics 


Number of sub-topics covered - Kinetics 


803 


0.31 


0.86 


0.0 


3.0 


topic_matter 


Number of sub-topics covered - Properties of Matter 


803 


0.74 


1.81 


0.0 


6.0 


topic_earth 


Number of sub-topics covered - Earth Systems 


803 


1.31 


2.91 


0.0 


9.0 


topic_elem_period 


Number of sub-topics covered - Elements and the Periodic System 


802 


0.39 


1.10 


0.0 


4.0 


topic_chemical_form 


Number of sub-topics covered - Chemical Formulas and Reactions 


801 


0.48 


1.38 


0.0 


5.0 


topic_acid_base 


Number of sub-topics covered - Acids, Bases, and Salts 


801 


0.55 


1.65 


0.0 


6.0 


topic_envi_chem 


Number of sub-topics covered - Environmental Chemistry 


801 


0.47 


1.42 


0.0 


5.0 


topic_nuclear_chem 


Number of sub-topics covered - Nuclear Chemistry 


801 


0.31 


1.05 


0.0 


4.0 


cover_nature_sc 


Science Content Covered - Nature of Science 


805 


0.38 


0.48 


0.0 


1.0 
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Appendix D1. Descriptive Statistics for PDAL Variabies: Based on Monthiy Log Levei Data (N=1,797) 



Variable 


Description 


N 


Mean 


SD 


Min 


Max 


cover_sc_tech 


Science Content Covered - Science and Technology 


805 


0.36 


0.48 


0.0 


1.0 


cover_health_env 


Science Content Covered - Science, Health and Environment 


805 


0.26 


0.44 


0.0 


1.0 


cover_measure_cal 


Science Content Covered - Measurement and Calculation in Science 


805 


0.34 


0.47 


0.0 


1.0 


cover_compo_live 


Science Content Covered - Components of Living Systems 


804 


0.19 


0.39 


0.0 


1.0 


cover_botany 


Science Content Covered - Botany 


804 


0.14 


0.35 


0.0 


1.0 


cover_animal_bio 


Science Content Covered - Animal Biology 


804 


0.14 


0.35 


0.0 


1.0 


cover_human_bio 


Science Content Covered - Human Biology 


804 


0.13 


0.34 


0.0 


1.0 


cover_ecology 


Science Content Covered - Ecology 


804 


0.16 


0.37 


0.0 


1.0 


cover_energy 


Science Content Covered - Energy 


803 


0.22 


0.41 


0.0 


1.0 


cover_motion_force 


Science Content Covered - Motion and Forces 


803 


0.17 


0.38 


0.0 


1.0 


cover_waves 


Science Content Covered - Characteristics and Behavior of Waves 


803 


0.13 


0.33 


0.0 


1.0 


cover_kinetics 


Science Content Covered - Kinetics 


803 


0.12 


0.33 


0.0 


1.0 


cover_matter 


Science Content Covered - Properties of Matter 


803 


0.17 


0.37 


0.0 


1.0 


cover_earth 


Science Content Covered - Earth Systems 


803 


0.20 


0.40 


0.0 


1.0 


cover_elem_period 


Science Content Covered - Elements and the Periodic System 


802 


0.12 


0.33 


0.0 


1.0 


cover_chemical_form 


Science Content Covered - Chemical Formulas and Reactions 


801 


0.12 


0.32 


0.0 


1.0 


cover_acid_base 


Science Content Covered - Acids, Bases, and Salts 


801 


0.11 


0.31 


0.0 


1.0 


cover_envi_chem 


Science Content Covered - Environmental Chemistry 


801 


0.11 


0.31 


0.0 


1.0 


cover_nuclear_chem 


Science Content Covered - Nuclear Chemistry 


801 


0.09 


0.28 


0.0 


1.0 


intensetopic_science 


Science topic intensity (contact hours divided by number of 1st-order topics) 


805 


4.84 


9.15 


0.0 


80.0 


Math Instructional Activities (dummy variables) 


IA1_math 


Number of reported 1st-order instructional activities (IA1) 


986 


2.85 


1.82 


0.0 


5.0 


Problem_Common 


Common Math Problem-Solving 


693 


0.67 


0.38 


0.0 


1.0 


Problem_Extended 


Extended Math Problem-Solving 


692 


0.39 


0.41 


0.0 


1.0 


M_Perform_Procedure 


Cognitive Demands: Perform Procedures 


823 


0.48 


0.34 


0.0 


1.0 


M_Make_Connection 


Cognitive Demands: Make Connections 


693 


0.62 


0.35 


0.0 


1.0 


Science Instructional Activities (dummy variables) 


IA1_science 


Number of reported 1st-order instructional activities 


797 


2.86 


1.89 


0.0 


5.0 


Lab_Work 


Laboratory-based Science Work 


511 


0.70 


0.34 


0.0 


1.0 


lnfo_Collect 


Science Data/Information Collection 


519 


0.72 


0.37 


0.0 


1.0 


S_Perform_Procedure 


Cognitive Demands: Perform Procedures 


602 


0.65 


0.36 


0.0 


1.0 


S_Make_Connection 


Cognitive Demands: Make Connections 


511 


0.61 


0.41 


0.0 


1.0 
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Appendix D2. Factors and Reliability of Professional Development Quality Constructs: 
Based on Monthly-log Level PDAL Data (N=1,797) 



Collective Participation 

Variable Label Factor 1 

i1490_1 I participated with most or all of the teachers from my school. 0.82 

i1490_2 I participated with most or all of the teachers from my department or grade level. 0.82 

alpha with all items 0.51 

Active Learning 

Variable Label Factor 1 

i1488_1 Observe demonstrations of teaching techniques? 0.50 

i1488_3 Lead group discussions? 0.70 

i1488_5 Develop curricula or lesson plans, which other participants or the activity leader reviewed? 0.75 

i1 488_6 Review student work or score assessments? 0.66 

i1488_7 Develop assessments or tasks as part of a formal professional development activity? 0.78 

i1488_8 Practice what you learned and receive feedback as part of a professional development activity 0.73 

i1488_9 Receive coaching or mentoring in the classroom? 0.61 

i1488_10 Give a lecture or presentation to colleagues? 0.69 

alpha with all items 0.83 

Coherence 

Variable Label Factor 1 

i1 489_1 Designed to support the school-wide improvement plan adopted by your school? 0.73 

i1489_2 Consistent with your department or grade level plan to improve teaching? 0.85 

i1 489_3 Consistent with your own goals for your professional development? 0.82 

i1489_4 Based explicitly on what you had learned in earlier professional development activities? 0.77 

i1489_5 Followed up with related activities that built upon what you learned as part of the activity? 0.82 

alpha with all items 0.85 
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Appendix D3. Descriptive Statistics for PDAL Variabies: Based on Teacher-ievei Data (N=273) 



Variable 


Description 


N 


Mean 


SD 


Min 


Max 


Activity Duration, Contact Hours, and Span 


MeanDur 


Duration (Mean) 


273 


1.85 


0.93 


0 


4 


MeanDDur 


Mean of Activity Duration dummy variable: 0=one day or less, 1= 2-4 days or longer 


273 


0.58 


0.36 


0 


1 


SumHours 


Total Contact Hours 


273 


101.14 


123.39 


1.0 


1060.0 


MeanHours 


Mean Contact Hours 


273 


16.89 


15.78 


1.0 


105.0 


Act_mo_N 


Average number of months 


273 


2.16 


1.79 


1.0 


11.2 


Act_Span 




273 


2.76 


2.44 


1.0 


14.0 


Act_Consecutive 




223 


0.67 


0.40 


0.0 


1.0 


meanAct_Hour307 


Mean Activity Contact Hour during July 2003 


150 


37.04 


29.60 


1.0 


150.0 


meanAct_Hour308 


Mean Activity Contact Hour during August 2003 


150 


14.85 


16.23 


1.0 


80.0 


meanAct_Hour309 


Mean Activity Contact Hour during September 2003 


117 


13.71 


19.99 


1.0 


160.0 


meanAct_Hour31 0 


Mean Activity Contact Hour during October 2003 


148 


12.62 


17.10 


1.0 


160.0 


meanAct_Hour31 1 


Mean Activity Contact Hour during November 2003 


117 


12.39 


18.52 


1.0 


160.0 


meanAct_Hour31 2 


Mean Activity Contact Hour during December 2003 


96 


10.66 


16.23 


1.0 


140.0 


meanAct_Hour401 


Mean Activity Contact Hour during January 2004 


98 


9.69 


17.26 


1.0 


160.0 


meanAct_Hour402 


Mean Activity Contact Hour during February 2004 


101 


11.46 


12.92 


1.0 


80.0 


meanAct_Hour403 


Mean Activity Contact Hour during March 2004 


96 


11.12 


11.09 


1.0 


60.0 


meanAct_Hour404 


Mean Activity Contact Hour during April 2004 


62 


13.25 


13.06 


1.5 


70.0 


meanAct_Hour405 


Mean Activity Contact Hour during May 2004 


74 


12.38 


10.19 


2.0 


60.0 


meanAct_Hour406 


Mean Activity Contact Hour during June 2004 


49 


27.00 


20.35 


1.0 


80.0 


meanAct_Hour407 


Mean Activity Contact Hour during July 2004 


49 


31.72 


19.58 


3.0 


80.0 


meanAct_Hour408 


Mean Activity Contact Hour during August 2004 


25 


23.01 


16.57 


2.5 


60.0 


sumAct_Hour307 


Sum Activity Contact Hour during July 2003 


150 


49.19 


40.50 


1.0 


180.0 


sumAct_Hour308 


Sum Activity Contact Hour during August 2003 


150 


19.84 


19.99 


1.0 


104.0 


sumAct_Hour309 


Sum Activity Contact Hour during September 2003 


117 


16.98 


22.13 


1.0 


160.0 


sumAct_Hour310 


Sum Activity Contact Hour during October 2003 


148 


16.07 


18.04 


1.0 


160.0 


sumAct_Hour31 1 


Sum Activity Contact Hour during November 2003 


117 


14.57 


19.46 


1.0 


160.0 


sumAct_Hour312 


Sum Activity Contact Hour during December 2003 


96 


12.66 


16.48 


1.0 


140.0 


sumAct_Hour401 


Sum Activity Contact Hour during January 2004 


98 


11.37 


17.42 


1.0 


160.0 


sumAct_Hour402 


Sum Activity Contact Hour during February 2004 


101 


14.88 


15.75 


1.0 


80.0 


sumAct_Hour403 


Sum Activity Contact Hour during March 2004 


96 


14.03 


14.12 


1.0 


80.0 


sumAct_Hour404 


Sum Activity Contact Hour during April 2004 


62 


16.56 


16.49 


2.0 


80.0 


sumAct_Hour405 


Sum Activity Contact Hour during May 2004 


74 


15.26 


12.92 


2.0 


62.0 


sumAct_Hour406 


Sum Activity Contact Hour during June 2004 


49 


29.90 


21.54 


1.0 


80.0 


sumAct_Hour407 


Sum Activity Contact Hour during July 2004 


49 


35.87 


21.96 


3.0 


82.0 


sumAct_Hour408 


Sum Activity Contact Hour during August 2004 


25 


25.16 


16.95 


2.5 


60.0 
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Appendix D3. Descriptive Statistics for PDAL Variabies: Based on Teacher-ievei Data (N=273) 



Variable 


Description 


N 


Mean 


SD 


Min 


Max 


PD Quaiity Measures 


ActiveLearn 


Active learning (mean score): 0=never, 1=rarely, 2=sometimes, 3=often 


270 


1.14 


0.60 


0.0 


3.0 


Coherence 


Coherence (mean score): 0=never, 1=rarely, 2=sometimes, 3=often 


268 


2.27 


0.50 


0.5 


3.0 


Collective 


Collective participation at a given month (sum score): On the scale of 0 - 2: number 
of postive response to two collective participation items 


269 


0.72 


0.58 


0.0 


2.0 


CollectiveSum 


Overall collective participation (sum score) 


269 


1.28 


0.78 


0.0 


2.0 


Content Focus Measures 


Purpose of PD 


Purp_Subject 


Strengthening subject matter knowledge 


262 


0.75 


0.31 


0.0 


1.0 


Math Topics 


topics_math 


Total number of 1st-order topics covered in the activity 


151 


2.27 


1.53 


0.0 


6.0 


intensetopic_math 


Math topic intensity (contact hours divided by number of 1st-order topics) 


151 


5.41 


8.55 


0.0 


80.0 


Math Instructionai Activities 


Problem_Common 


Common math exercise/problem-solving: estimate, predict, apply, analyze, infer 


139 


0.67 


0.26 


0.0 


1.0 


Problem_Extended 


Extended problem-solving 


139 


0.38 


0.30 


0.0 


1.0 


M_Perform_Procedure Perform procedures (math) 


142 


0.49 


0.24 


0.0 


1.0 


M_Make_Connection 


Make connections (math) 


139 


0.62 


0.24 


0.0 


1.0 


IA1_math 


Number of broad instructional activities focused 


151 


3.02 


1.29 


0.0 


5.0 


IA2_math 


Number of specific instructional strategies focused within broad instructional 
activities covered 


143 


13.64 


7.48 


0.0 


35.0 


Science Topics 


topics_science 


Total number of 1st-order topics covered in the activity 


125 


5.00 


4.28 


0.0 


24.5 


intensetopic_science 


Science topic intensity (contact hours divided by number of 1st-order topics) 


125 


5.06 


6.24 


0.0 


36.3 


Science Instructionai Activities 


Lab_Work 


Lab-based activities 


111 


0.71 


0.25 


0.0 


1.0 


lnfo_Collect 


Collecting science data/info outside lab 


112 


0.70 


0.27 


0.0 


1.0 


S_Perform_Procedure 


Perform procedures (science) 


114 


0.66 


0.24 


0.0 


1.0 


S_Make_Connection 


Make connections (science) 


111 


0.62 


0.30 


0.0 


1.0 


IA1_science 


Number of broad instructional activities focused 


122 


3.00 


1.28 


0.0 


5.0 


IA2_science 


Number of specific instructional strategies focused within broad instructional 
activities covered 


119 


14.80 


8.20 


0.0 


34.0 



85 







Appendix D4. Factors and Reliability of Professional Development Quality Constructs: 
Based on Teacher-level PDAL Data (N=273) 



Collective Participation In a Given Month 



Variable 


Label 


Factor 1 




i1490_1 


I participated with most or all of the teachers from my school. 






i1490_2 


I participated with most or all of the teachers from my department or grade level. 








alpha with all items 


0.54 




Overall Collective Participation 






Variable 


Label 


Factor 1 




collect_sch 


I participated with most or all of the teachers from my school. 






collect_dept 


I participated with most or all of the teachers from my department or grade level. 








alpha with all items 


0.60 




Active Learning 








Variable 


Label 


Factor 1 


Factor 2 


i1488_1 


Observe demonstrations of teaching techniques? 


-0.02 


0.86 


i1488_3 


Lead group discussions? 


0.51 


0.56 


i1488_5 


Develop curricula or lesson plans, which other participants or the activity leader reviewed? 


0.80 


0.22 


i1488_6 


Review student work or score assessments? 


0.83 


0.05 


i1488_7 


Develop assessments or tasks as part of a formal professional development activity? 


0.84 


0.23 


i1488_8 


Practice what you learned and receive feedback as part of a professional development activit 


0.50 


0.57 


i1488_9 


Receive coaching or mentoring in the classroom? 


0.21 


0.68 


i1488_10 


Give a lecture or presentation to colleagues? 


0.53 


0.50 




alpha with all items 


0.85 




Coherence 








Variable 


Label 


Factor 1 




i1489_1 


Designed to support the school-wide improvement plan adopted by your school? 


0.73 




i1489_2 


Consistent with your department or grade level plan to improve teaching? 


0.86 




i1489_3 


Consistent with your own goals for your professional development? 


0.84 




i1489_4 


Based explicitly on what you had learned in earlier professional development activities? 


0.69 




i1489_5 


Followed up with related activities that built upon what you learned as part of the activity? 


0.82 






alpha with all items 


0.83 
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Appendix D4. Factors and Reliability of Professional Development Quality Constructs: 
Based on Teacher-level PDAL Data (N=273) 



Common Math Problem-Solving 


Variable 


Label 


Factor 1 




i1691_3 


Students Work Individually - Explain their reasoning or thinking in solving a problem, using 


0.58 






several sentences orally or in writing 






i1691_4 


Students Work Individually - Apply mathematical concepts to real-world problems 


0.72 




i1691_5 


Students Work Individually - Make estimates, predictions or hypotheses 


0.86 




i1691_6 


Students Work Individually - Analyze data to make inferences or draw conclusions 


0.86 




i2160_3 


Students Work in Pairs or Small Groups - Talk about their reasoning or thinking in solving a 


0.79 






problem 






i2160_4 


Students Work in Pairs or Small Groups - Apply mathematical concepts to real-world 


0.80 






problems 






12160 5 


Students Work in Pairs or Small Groups - Make estimates, predictions or hypotheses 


0.89 




I2160 6 


Students Work in Pairs or Small Groups - Analyze data to make inferences or draw 


0.85 






conclusions 








alpha with all items 


0.91 




Extended Math Problem-Solving 






Variable 


Label 


Factor 1 


Factor 2 


i1691_7 




0.94 


0.16 




Students Work Individually - Work on a problem that takes at least 45 minutes to solve 






i1691_8 


Students Work Individually - Complete or conduct proofs or demonstrations of their 


0.23 


0.89 




mathematical reasoning 






i2160_7 


Students Work in Pairs or Small Groups - Work on a problem that takes at least 45 minutes 


0.91 


0.24 




to solve 






12160 8 


Students Work in Pairs or Small Groups - Complete or conduct proofs or demonstrations of 


0.16 


0.91 




their mathematical reasoning 








alpha with all items 


0.80 
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Appendix D4. Factors and Reliability of Professional Development Quality Constructs: 
Based on Teacher-level PDAL Data (N=273) 



Laboratory-based Work 



Variable 


Label 


Factor 1 




i2085_1 


Lab Activities, Investigations, or Experiments - Make educated guesses, predictions, or 


0.55 


0.55 




hypotheses 






i2085_2 


Lab Activities, Investigations, or Experiments - Follow step-by-step directions 


0.64 


0.10 


i2085_3 


Lab Activities, Investigations, or Experiments - Use science equipment or measuring tools 


0.70 


0.12 


i2085_4 


Lab Activities, Investigations, or Experiments - Collect data 


0.70 


0.29 


i2085_5 


Lab Activities, Investigations, or Experiments - Change a variable in an experiment to test a 


0.25 


0.77 




hypothesis 






i2085_6 


Lab Activities, Investigations, or Experiments - Organize and display information in tables or 


0.24 


0.69 




graphs 






i2085_7 


Lab Activities, Investigations, or Experiments - Analyze and interpret science data 


0.66 


0.45 


i2085_8 


Lab Activities, Investigations, or Experiments - Design their own investigation or experiment 


0.12 


0.83 




to solve a scientific question 






i2085_9 


Lab Activities, Investigations, or Experiments - Make observations/classifications 


0.80 


0.29 



alpha with all items 0.86 

Collecting Science Data/Information Outside the Lab 

Variable Label Factor 1 

i2087_1 Collecting Science Data or Information - Have class discussions about the data 0.87 

i2087_2 Collecting Science Data or Information - Organize and display the information in tables or 0.69 

graphs 

i2087_3 Collecting Science Data or Information - Make a prediction based on the data 0.67 

i2087_4 Collecting Science Data or Information - Analyze and interpret the information or data, orally 0.86 

or in writing 

i2087_7 Collecting Science Data or Information - Make a presentation to the class on the data, 0.59 

analysis or interpretation 

alpha with all items 0.78 
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Appendix D5. Quality of Math Professional Development Activities Compared by MSP Site and Treatment Status: 

Based on Teacher-level PDAL Data (N = 148) 



Variable Category 




% Duration 
>2-4 days 


Mean 

contact 

hours 


Mean 

activity 

span 


Active 

learning 


Coherence 


Collective 

participation 

(Overall) 


Collective 
participation 
(At a given 
month) 


MSP Site 




*** 


** 


*** 


ns 


* 


ns 


ns 


Brockport 


37 


0.67 


22.5 


3.3 




2.4 






Cleveland 


49 


0.73 


13.9 


3.7 




2.2 






Corpus Christi 


34 


0.40 


14.0 


2.1 




2.3 






El Paso 


28 


0.46 


11.9 


1.5 




2.0 






Treatment 




** 


** 


** 


ns 


ns 


ns 


* 


Comparison 


45 


0.46 


11.2 


2.1 








0.86 


Treatment 


103 


0.64 


17.7 


3.1 








0.62 


MSP_Site*T reatment 




ns 


* 


ns 


ns 


ns 


ns 


ns 


Brockport-Comparison 


12 




10.7 












Brockport-T reatment 


25 




28.2 












Cleveland-Comparison 


14 




11.3 












Cleveland-T reatment 


35 




15.0 












Corpus Christi-Comparison 


6 




18.2 












Corpus Christi-Treatment 


28 




13.1 












El Paso-Comparison 


13 




8.4 












El Paso-T reatment 


15 




14.9 













Note: ^ N may vary due to variable-wise missing data. 

ns denotes non-significance, *<.05, **<.01, and ***<.001. 
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Appendix D5. Quality of Math Professional Development Activities Compared by MSP Site and Treatment Status: 

Based on Teacher-level PDAL Data (N = 148) 



Variable Category 




Strengthen 
subject matter 
knowledge 


Number of 
topical 
areas 
focused 


Topic 

intensity 


Common 

math 

problem 

solving 


Extended 

math 

problem 

solving 


Perform 

procedures 


Make 

connections 


Number of 
instructional 
activities 
focused 


MSP Site 




ns 


* 


ns 


* 


ns 


ns 


ns 


ns 


Brockport 


37 




1.9 




0.58 










Cleveland 


49 




2.2 




0.64 










Corpus Christ! 


34 




2.9 




0.76 










El Paso 


28 




2.0 




0.74 










Treatment 




ns 


ns 


* 


ns 


ns 


ns 


ns 


ns 


Comparison 


45 






2.5 












Treatment 


103 






6.7 












MSP Site*Treatment 




ns 


ns 


ns 


ns 


ns 


ns 


ns 


ns 



Brockport-Comparison 12 

Brockport-Treatment 25 

Cleveland-Comparison 14 

Cleveland-Treatment 35 

Corpus Christi-Comparison 6 

Corpus Christi-Treatment 28 

El Paso-Comparison 13 

El Paso-Treatment 15 

Note: ^ N may vary due to variable-wise rr 

ns denotes non-significance, *<.05, 
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Appendix D6. Quality of Science Professional Development Activities Compared by MSP Site and Treatment Status: 

Based on Teacher-level PDAL Data (N = 125) 



Collective 









Mean 


Mean 






Collective 


participation 






% Duration 


contact 


activity 


Active 




participation 


(At a given 


Variable Category 




>2-4 days 


hours 


span 


learning 


Coherence 


(Overall) 


month) 


MSP Site 




*** 


ns 


*** 


ns 


ns 


ns 


ns 


Brockport 


18 


0.69 




2.2 










Cleveland 


63 


0.66 




3.6 










Corpus Christ! 


23 


0.39 




1.6 










El Paso 


21 


0.40 




1.7 










Treatment 




* 


ns 


* 


ns 


ns 


ns 


ns 


Comparison 


54 


0.48 




2.2 










Treatment 


71 


0.64 




3.1 










MSP Site*Treatment 




ns 


ns 


ns 


ns 


ns 


ns 


ns 



Brockport-Comparison 


8 


Brockport-T reatment 


10 


Cleveland-Comparison 


15 


Cleveland-T reatment 


48 


Corpus Christi-Comparison 


22 


Corpus Christi-T reatment 


1 


El Paso-Comparison 


9 


El Paso-Treatment 


12 



Note: ^ N may vary due to variable-wise missing data. 

ns denotes non-significance, *<.05, **<.01 , and ***<.001 . 
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Appendix D6. Quality of Science Professional Development Activities Compared by MSP Site and Treatment Status: 

Based on Teacher-level PDAL Data (N = 125) 



Variable Category 




Strengthen 
subject matter 
knowledge 


Number of 
topical 
areas 
focused 


Topic 

intensity 


Lab-based 

work 


Science 

data/ 

information 

collection 


Peform 

procedures 


Make 

connections 


Number of 
instructional 
activities 
focused 


MSP Site 




ns 


ns 


** 


* 


** 


* 


* 


ns 


Brockport 


18 






9.4 


0.55 


0.53 


0.53 


0.48 




Cleveland 


63 






4.9 


0.77 


0.76 


0.71 


0.69 




Corpus Christi 


23 






3.6 


0.71 


0.70 


0.64 


0.57 




El Paso 


21 






3.5 


0.67 


0.64 


0.60 


0.54 




Treatment 




*** 


ns 


* 


** 


*** 


* 


** 


*** 


Comparison 


54 


0.61 




3.6 


0.64 


0.61 


0.60 


0.52 


2.5 


Treatment 


71 


0.81 




6.2 


0.76 


0.77 


0.69 


0.68 


3.3 


MSP_Site*T reatment 




ns 


* 


* 


ns 


ns 


ns 


ns 


ns 


Brockport-Comparison 


8 




7.0 


3.6 












Brockport-T reatment 


10 




4.5 


14.0 












Cleveland-Comparison 


15 




5.3 


4.0 












Cleveland-T reatment 


48 




4.4 


5.2 












Corpus Christi-Comparison 


22 




5.0 


3.7 












Corpus Christi-Treatment 


1 




10.7 


0.9 












El Paso-Comparison 


9 




3.0 


2.9 












El Paso-T reatment 


12 




7.1 


3.9 













Note: ^ N may vary due to variable-wise rr 

ns denotes non-significance, *<.05, 
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Appendix El. Amount of Teachers' Instructional Activity Using Technology (Based on Year 1 SEC Data): 

By Subject Taught and By MSP Program Treatment Status 





Comparison 




Treatment 








Mathematics 


Mean ^ 


SD 


N 


Mean ^ 


SD 


N 


F-value 


2 

P 


How much of the total math instructional time do students in the target 
class use computers, calculators, or other technology to learn math?” 


3.45 


1.62 


19 


3.05 


1.34 


28 


0.83 


ns 


When students in the target class are engaged in activities that involve the use of calculators, computers, or other educational technology as part 


of mathematics instruction, how much time do they: 


















Learn facts 


2.08 


1.55 


19 


2.02 


1.74 


28 


0.02 


ns 


Practice procedures 


2.63 


1.50 


19 


2.89 


1.68 


28 


0.30 


ns 


Use sensors and probes 


0.53 


1.29 


19 


0.66 


1.19 


28 


0.14 


ns 


Retrieve or exchange data or information (for example, using the 
Internet or partnering with another class) 


0.74 


1.39 


19 


1.70 


1.70 


28 


4.16 


* 


Display and analyze data 


1.92 


1.85 


19 


2.27 


1.69 


28 


0.44 


ns 


Develop geometric concepts (for example, using simulations) 


0.95 


1.50 


19 


1.39 


1.54 


28 


0.97 


ns 




Comparison 




Treatment 








Science 


Mean ^ 


SD 


N 


Mean ^ 


SD 


N 


F-value 


2 

P 


How much of the total science instructional time do students in the 
target class use computers, calculators, or other technology to learn 
science? 


2.35 


1.73 


17 


2.29 


1.44 


14 


0.91 


ns 


When students in the target class are engaged in activities that involve the use of calculators, computers, or other educational technology as part 


of science instruction, how much time do they: 


















Learn facts 


3.12 


1.36 


17 


3.07 


0.92 


14 


0.01 


ns 


Practice procedures 


3.12 


1.32 


17 


2.57 


1.34 


14 


1.30 


ns 


Use sensors and probes (for example, CBL's) 


1.06 


1.25 


17 


0.21 


0.58 


14 


5.42 


* 


Retrieve or exchange data or information (for example, using the 
Internet or partnering with another class) 


1.53 


1.46 


17 


1.79 


1.58 


14 


0.22 


ns 


Display and analyze data 


2.00 


1.34 


17 


2.08 


1.59 


14 


0.02 


ns 


Develop problems using simulations 


1.35 


1.46 


17 


1.57 


1.40 


14 


0.18 


ns 



Note: * The amount of instractional activity was measured on the scale of 0=none, l=little, 2=some, 3=moderate, 4=considerable, and 5=almost all. 
^ ns stands for non-signficance, * <.05, ** <.01, and *** <.001. 
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Appendix E2. Teachers* Professional Development Experience in the Use of Technology (Based on PDAL Data): 

By Subject Taught and By MSP Program Treatment Status 





Comparison 




Treatment 








Mathematics 


Mean ^ 


SD 


N 


Mean ^ 


SD 


N 


F-value 


2 

P 


Was an instructional topic in the “use of computers, calculators, or 
other technology to learn math” covered in this professional 
development activity? 


0.64 


0.34 


12 


0.78 


0.22 


25 


2.29 


ns 


Did the professional development focus on any of the following instructional strategies for use in your classroom with your students (relating to 


calculators, computers, or other educational technology)? 


















Learn facts 


0.46 


0.40 


11 


0.38 


0.33 


25 


0.38 


ns 


Practice procedures 


0.60 


0.32 


11 


0.55 


0.35 


25 


0.13 


ns 


Use sensors and probes 


0.05 


0.15 


11 


0.32 


0.26 


25 


10.68 


** 


Retrieve or exchange data or information (for example, using the 
Internet or partnering with another class) 


0.17 


0.25 


11 


0.43 


0.29 


25 


6.74 


* 


Display and analyze data 


0.53 


0.37 


11 


0.76 


0.27 


25 


4.26 


* 


Develop geometric concepts (for example, using simulations) 


0.11 


0.20 


11 


0.47 


0.34 


25 


10.96 


** 




Comparison 




Treatment 








Science 


Mean ^ 


SD 


N 


Mean ^ 


SD 


N 


F-value 


2 

P 


Was an instructional topic in the “use of computers, calculators, or 
other technology to learn science” covered in this professional 
development activity? 


0.46 


0.41 


8 


0.71 


0.32 


9 


1.92 


ns 


Did the professional development focus on any of the following instructional strategies for use in 


your classroom with your students (relating to 


calculators, computers, or other educational technology)? 


















Learn facts 


0.60 


0.37 


6 


0.56 


0.41 


9 


0.03 


ns 


Practice procedures 


0.63 


0.31 


6 


0.55 


0.40 


9 


0.18 


ns 


Use sensors and probes (for example, CBL's) 


0.07 


0.10 


6 


0.54 


0.38 


9 


8.68 


* 


Retrieve or exchange data or information (for example, using the 
Internet or partnering with another class) 


0.72 


0.34 


6 


0.69 


0.31 


9 


0.03 


ns 


Display and analyze data 


0.70 


0.27 


6 


0.69 


0.34 


9 


0.00 


ns 


Develop problems using simulations 


0.10 


0.17 


6 


0.77 


0.24 


9 


34.67 


*** 



Note: ^ The experience of professional development was measured on the scale of 0=no and l=yes. 
^ ns stands for non-signficance, * <.05, ** <.01, and *** <.001. 
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Appendix F: Summary of Results of Hypothesis Tests: Math 





Comparisons among 


Treatment vs. 


Math 


MSP Program Sites 


Comparison Teachers 




Test of Hypothesis: 


Test of Hypothesis: 




1 3 


2 4 


Cluster of Variables 


SEC1^ PDAL^ SEC2^ 


SEC1^ PDAL^ SEC2^ 



Teacher/Class/School Characteristics 

Course Type 

Number of Students in Target Class 

Achievement Levels of Students(Teacher perception 

Number of Years in Teaching 

Number of Years in Teaching at This School 

Highest Degree Held 

Relational Trust 



ns 






ns 






** 






ns 






ns 






*** 






*** 







Instructional Activities/Strategies 

Assessment Use 

Instructional Influece 

Cognitive Demands: Memorize 

Cognitive Demands: Perform Procedures 

Cognitive Demands: Communicate Understanding 

Cognitive Demands: Conjecture, Generalize, Prove 

Cognitive Demands: Make Connections 



* 






*** 






ns 






ns 






ns 






ns 






ns 







Content of Classroom Instruction 

Instructional Alignment with Standards 

Instructional Alignment with Assessment 

Cognitive Demands: Memorize 

Cognitive Demands: Perform Procedures 

Cognitive Demands: Communicate Understanding 

Cognitive Demands: Conjecture, Generalize, Prove 

Cognitive Demands: Make Connections 

Breadth of Content Coverage 

Topic Converage: Number Sense/Properties/Relatic 

Topic Converage: Operations 

Topic Converage: Measurement 

Topic Converage: Algebraic Concepts 

Topic Converage: Geometric Concepts 

Topic Converage: Data Analysis/Probability/Statistic 

Topic Converage: Instructional Technology 



*** 






* 






ns 






ns 






ns 






* 






ns 






*** 






ns 






ns 






ns 






ns 






ns 






ns 






* 







ns 






ns 






ns 






* 






* 






ns 






ns 







ns 






ns 






ns 






* 






ns 






ns 






* 







ns 






ns 






ns 






ns 






ns 






ns 






ns 






ns 






ns 






ns 






ns 






ns 






* 






ns 






ns 







Professional Development 

Frequency 

% Duration >2-4 days 
Mean Contact Hours 
Activity Span 
Active Learning 
Coherence 

Collective participation (Overall) 

Collective participation (At a given month) 
Strengthen subject matter knowledge 
Number of topical areas focused 
Topic Intensity 



ns 








ns 








*** 






** 




ns 


** 




ns 


** 






*** 






** 




* 


ns 




ns 


ns 




ns 


* 




ns 


ns 




*** 


ns 




ns 


ns 






ns 






* 






ns 






ns 






* 






ns 






ns 








* 
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Appendix F: Summary of Results of Hypothesis Tests: Math 



Cluster of Variables 



1 3 

SEC1^ PDAL^ SEC2^ 



2 4 

SEC1^ PDAL^ SEC2^ 





* 








ns 






ns 






ns 






ns 






ns 






ns 






ns 






ns 






ns 




*** 






ns 






*** 






ns 






ns 






ns 






** 






ns 






*** 






ns 







Common Math Problem-Solving 
Extended Math Problem-solving 
Perform Procedures 
Make Connections 

Number of instructional activities focused 

In-depth study emphasis (single item) 

Content Emphasis 

Data Emphasis 

Student Learning Emphasis 

Standards & Instruction Emphasis 



Note: ' N of math teachers who responded to SEC1 (Year 1 Survey of Enacted Curriculum) is 210. 

^ N of math teachers who completed the PDAL (Professional Development Activity Log) is 138. 
^ SEC2 measures will be available in spring of 2005. 

Shaded areas represent not-applicable or non-existent measures, 
ns denotes not-significance, * p<.05, ** p<.01 , and *** p<.001 . 
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Appendix F: Summary of Results of Hypothesis Tests: Science 





Comparisons between 


Treatment vs. 


Science 


MSP Program Sites 


Comparison Teachers 




Test of Hypothesis: 


Test of Hypothesis: 




1 3 


2 4 


Cluster of Variables 


SEC1^ PDAL^ SEC2^ 


SECV PDAL^ SEC2^ 



Teacher/Class/School Characteristics 



Course Type 


* 






Number of Students in Target Class 


ns 






Achievement Levels of Students(Teacher perceptio 


* 






Number of Years in Teaching 


ns 






Number of Years in Teaching at This School 


ns 






Highest Degree Held 


*** 






Relational Trust 


* 






Instructional Activities/Strategies 






Assessment Use 


*** 






Instructional Influece 


ns 






Cognitive Demands: Memorize 


ns 






Cognitive Demands: Perform Procedures 


ns 






Cognitive Demands: Communicate Understanding 


ns 






Cognitive Demands: Analyze Information 


ns 






Cognitive Demands: Make Connections 


* 






Content of Classroom Instruction 






Instructional Alignment with Standards 


*** 






Instructional Alignment with Assessment 


*** 






Cognitive Demands: Memorize 


ns 






Cognitive Demands: Perform Procedures 


ns 






Cognitive Demands: Communicate Understanding 


ns 






Cognitive Demands: Analyze Information 


ns 






Cognitive Demands: Make Connections 


ns 






Breadth of Content Coverage 


ns 






Topic Converage 


ns 






Professional Development 






Frequency 


* 






% Duration >2-4 days 




*** 




Mean Contact Hours 


** 


ns 




Activity Span 




*** 




Active Learning 


* 


ns 




Coherence 


ns 


ns 




Collective participation (Overall) 


ns 


ns 




Collective participation (At a given month) 




ns 




Strengthen subject matter knowledge 




ns 




Number of topical areas focused 




ns 




Topic Intensity 




** 




Laboratory Work 




* 




Science Data/Information Collection 




** 




Perform Procedures 




* 




Make Connections 




* 




Number of instructional activities focused 




ns 




In-depth study emphasis (single item) 


ns 







ns 






ns 






* 






ns 






ns 






ns 






ns 







ns 






ns 






ns 






* 






ns 






ns 






ns 







ns 






ns 






ns 






* 






ns 






* 






* 






ns 






ns 







** 








* 




*** 


ns 






* 




** 


ns 




** 


ns 




ns 


ns 






ns 






*** 






ns 






* 






** 






*** 






* 






** 






*** 




ns 
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Appendix F: Summary of Results of Hypothesis Tests: Science 



Science 



Comparisons between Treatment vs. 

MSP Program Sites Comparison Teachers 



Cluster of Variables 



Test of Hypothesis: 

1 3 

SEC1^ PDAL^ SEC2^ 



Test of Hypothesis: 



2 


4 


SECV 


PDAL^ 


SEC2^ 


* 






ns 






ns 






ns 







ns 






ns 






ns 






ns 







Content Emphasis 
Data Emphasis 
Student Learning Emphasis 
Standards & Instruction Emphasis 



Note: ^ N of science teachers who responded to SEC1 (Year 1 Survey of Enacted Curriculum) is 180. 

^ N of science teachers who completed the PDAL (Professional Development Activity Log) is 1 1 2. 
® SEC2 measures will be available in spring of 2005. 

"^Some significant effects were found in some of the middle- and high school science topics 
Shaded areas represent not-applicable or non-existent measures, 
ns denotes not-significance, * p<.05, ** p<.01 , and *** p<.001 . 
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Appendix G 



Alignment Index: 0.21 

Re-centered 0.35 

State I Gr. 7 Mathematics Assessment (2003) State I Gr. 7 Mathematics Standards (2003) 



Aii Content Areas 



All Content Areas 




Number Sense 
Operations 
Measurement 
Algebraic Concepts 
Geometric Concepts 
Data Analysis 
Instructional Technology 




□ -0.01 8-0.002 □ 0.002-0.022 0 0.022-0.042 0 0.042-0.062 0 0.062-0.082 
00.082-0.102 BO.102-0.122 00.122-0.142 00.142-0.162 00.162-0.182 



Number Sense 
Operations 



Measurement 



Algebraic Concepts 



Geometric Concepts 
Data Analysis 



Instructional Technology 



0-0.018-0.002 0 0.002-0.022 0 0.022-0.042 0 0.042-0.062 0 0.062-0.082 
00.082-0.102 00.102-0.122 00.122-0.142 00.142-0.162 00.162-0.182 




Contour Interval = 2% 

Re-centered Alignment uses aggregated data for each content area to perform calculation. 



99 



Appendix G 



State I Gr. 7 Mathematics Assessment (2003) Alignment index: 0.21 State I Gr. 7 Mathematics Standards (2003) 

Data Analysis, Probability, Statistics Re-Centered 0.14 pg^g Analysis, Probability, Statistics 




^ ^ o o o 



Bar graph, histogram 

Pie charts, circle graphs 

Pictographs 

Line graphs 

Stem and Leaf plots 

Scatter plots 

Box plots 

Mean, median, mode 
Line of best fit 
Quartiles, percentiles 
Sampling, Sample spaces 
Simple probability 
Compound probability 

Comb./permutations 

Summarize data 
(table/graph) 



4 .^ 



% ® 



□ -0.001 8-0.0002 


□ 0.0002-0.0022 


□ 0.0022-0.0042 


□ 0.0042-0.0062 


□ 0.0062-0.0082 





Bar graph, histogram 

Pie charts, circle graphs 

Pictographs 

Line graphs 

Stem and Leaf plots 

Scatter plots 

Box plots 

Mean, median, mode 
Line of best fit 
Quartiles, percentiles 
Sampling, Sample spaces 
Simple probability 
Compound probability 

Comb./permutations 

Summarize data 
(table/graph) 




□ -0.001 8-0.0002 


□ 0.0002-0.0022 


□ 0.0022-0.0042 


□ 0.0042-0.0062 


□ 0.0062-0.0082 






Contour Interval = 0.2% 

Re-centered adjusts data to reflect this content area only. 
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Appendix G 



Alignment Index: 0.37 

Re-centered 0.70 

State X Gr. 8 Mathematics Assessment (2003) State X Gr. 8 Mathematics Standards (2003) 



Ail Content Areas 



All Content Areas 




Number Sense 
Operations 
Measurement 
Algebraic Concepts 
Geometric Concepts 

Data Analysis 

Instructional 

Technology 




□ -0.01 8-0.002 □ 0.002-0.022 0 0.022-0.042 0 0.042-0.062 0 0.062-0.082 
00.082-0.102 aO.102-0.122 EO.122-0.142 *0.142-0.162 ■0.162-0.182 




Number Sense 



Operations 



Measurement 



Algebraic Concepts 
Geometric Concepts 

Data Analysis 

Instructional 

Technology 




0-0.018-0.002 0 0.002-0.022 0 0.022-0.042 0 0.042-0.062 0 0.062-0.082 
00.082-0.102 BO.102-0.122 00.122-0.142 BO.142-0.162 BO.162-0.182 



Contour Interval = 2% 

Re-centered Alignment uses aggregated data for each content area to perform calculation. 
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Appendix G 



State X Gr. 8 Mathematics Assessment (2003) Alignment Index: 0.37 State X Gr. 8 Mathematics Standards (2003) 

Data Analysis, Probability, Statistics Re-Centered 0.41 Analysis, Probability, Statistics 




Bar graph, histogram 

Pie charts, circle graphs 

Pictographs 

Line graphs 

Stem and Leaf plots 

Scatter plots 

Box plots 

Mean, median, mode 
Line of best fit 
Quartiles, percentiles 
Sampling, Sample spaces 
Simple probability 
Compound probability 

Comb./permutations 

Summarize data 
(table/graph) 




□ -0.001 8-0.0002 


□ 0.0002-0.0022 


□ 0.0022-0.0042 


□ 0.0042-0.0062 


□ 0.0062-0.0082 





Bar graph, histogram 

Pie charts, circle graphs 

Pictographs 

Line graphs 

Stem and Leaf plots 

Scatter plots 

Box plots 

Mean, median, mode 
Line of best fit 
Quartiles, percentiles 
Sampling, Sample spaces 
Simple probability 
Compound probability 

Comb./permutations 

Summarize data 
(table/graph) 




□ -0.001 8-0.0002 


□ 0.0002-0.0022 


□ 0.0022-0.0042 


□ 0.0042-0.0062 


□ 0.0062-0.0082 






Contour Interval = 0.2% 

Re-centered adjusts data to reflect this content area only. 
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Appendix G 



State Y Grade 8 Mathematics Assessment (2003) 



Alignment Index: 
Re-centered 



0.39 

0.64 

State Y Intermediate Mathematics Standards (2003) 



All Content Areas 



All Content Areas 




Number Sense 
Operations 
Measurement 
Algebraic Concepts 
Geometric Concepts 
Data Analysis 
Instructional Technology 




□ -0.01 8-0.002 □ 0.002-0.022 0 0.022-0.042 0 0.042-0.062 0 0.062-0.082 
00.082-0.102 BO.102-0.122 00.122-0.142 00.142-0.162 00.162-0.182 




Number Sense 



Operations 



Measurement 



Algebraic Concepts 



Geometric Concepts 



Data Analysis 



Instructional Technology 




0-0.018-0.002 0 0.002-0.022 0 0.022-0.042 0 0.042-0.062 0 0.062-0.082 
00.082-0.102 00.102-0.122 00.122-0.142 00.142-0.162 00.162-0.182 



Contour Interval = 2% 

Re-centered Alignment uses aggregated data for each content area to perform calculation. 
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Appendix G 



State Y Grade 8 Mathematics Assessment (2003) Alignment index: 0.39 State Y Intermediate Mathematics Standards (2003) 

Data Analysis, Probability, Statistics Re-Centered o.42 pg^g Analysis, Probability, Statistics 



Bar graph, histogram 

Pie charts, circle graphs 

Pictographs 

Line graphs 

Stem and Leaf plots 

Scatter plots 

Box plots 

Mean, median, mode 
Line of best fit 
Quartiles, percentiles 
Sampling, Sample spaces 
Simple probability 
Compound probability 

Comb./permutations 

Summarize data 
(table/graph) 




□ -0.001 8-0.0002 


□ 0.0002-0.0022 


□ 0.0022-0.0042 


□ 0.0042-0.0062 


□ 0.0062-0.0082 






Bar graph, histogram 

Pie charts, circle graphs 

Pictographs 

Line graphs 

Stem and Leaf plots 

Scatter plots 

Box plots 

Mean, median, mode 
Line of best fit 
Quartiles, percentiles 
Sampling, Sample spaces 
Simple probability 
Compound probability 

Comb./permutations 

Summarize data 
(table/graph) 




□ -0.001 8-0.0002 


□ 0.0002-0.0022 


□ 0.0022-0.0042 


□ 0.0042-0.0062 


□ 0.0062-0.0082 















































































rn 










LJ 


















1 










/ 


7 — ^ 


Tl 
































:rL 




\ 










■^ / 



Contour Interval = 0.2% 

Re-centered adjusts data to reflect this content area only. 



104 



Appendix G 



State I Grade 7 Science Assessment (2003) Alignment index: 
All Content Areas Re-centered 




Nature of Science 
Science & Technology 
Sci., Health, & Envrnmnt. 
Measurement in Science 
Components of Living Sys. 
Botany 

Animal Biology 
Human Biology 
Evolution 

Reproduction & Devipmnt. 

Ecology 

Energy 

Motion & Forces 
Electricity 

Characteristics of Waves 
Kinetics 

Properties of Matter 
Earth Systems 
Astronomy 
Meteorology 

Elements & The Per. Sys. 
Chem. Formulas & Reactions 
Acids, Bases, & Salts 
Environmental Chemistry 
Nuclear Chemistry 






% 



0.16 

0.29 



State I Grade 7 Science Standards (2003) 
All Content Areas 




Nature of Science 
Science & Technology 
Sci., Health, & Envrnmnt. 
Measurement in Science 
Components of Living Sys. 
Botany 

Animal Biology 
Human Biology 
Evolution 

Reproduction & Devipmnt. 

Ecology 

Energy 

Motion & Forces 
Electricity 

Characteristics of Waves 
Kinetics 

Properties of Matter 
Earth Systems 
Astronomy 
Meteorology 

Elements & The Per. Sys. 
Chem. Formulas & Reactions 
Acids, Bases, & Salts 
Environmental Chemistry 
Nuclear Chemistry 









□ -0.01 8-0.002 □ 0.002-0.022 □ 0.022-0.042 0 0.042-0.062 0 0.062-0.082 
00.082-0.102 HO.102-0.122 0 0.122-0.142 BO.142-0.162 BO.162-0.182 



0-0.018-0.002 0 0.002-0.022 0 0.022-0.042 0 0.042-0.062 0 0.062-0.082 
00.082-0.102 HO.102-0.122 SO.122-0.142 BO.142-0.162 BO.162-0.182 



Contour Interval = 2% 

Re-centered Alignment uses aggregated data for each content area to perform calculation. 
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Appendix G 



State I Grade 7 Science Assessment (2003) 



Alignment Index: 
Re-Centered 




The International System 

Mass & Weight 

Length 

Volume 

Time 

Temperature 
Accuracy & Precision 
Significant Digits 
Derived Units 
Conversion Factors 
Density 
Data Displays 



Vx\\> ' 












□ -0.0048-0.0002 □ 0.0002-0.0052 □ 0.0052-0.01 02 El 0.01 02-0.01 52 



0.16 

0.26 



State I Grade 7 Science Standards (2003) 



The International System 

Mass & Weight 

Length 

Volume 

Time 

Temperature 
Accuracy & Precision 
Significant Digits 
Derived Units 
Conversion Factors 
Density 
Data Displays 



□ -0.0048-0.0002 □ 0.0002-0.0052 □ 0.0052-0.01 02 □ 0.01 02-0.01 52 



Measurement in Science 









^ O ^ ^ 



% \ V V % 



% 



Contour Interval = 0.5% 

Re-centered adjusts data to reflect this content area only. 
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